Wednesday, April 08, 2009

Clojure on Google AppEngine

Note: This post is quite old, and Compojure has changed (and gotten quite a bit bigger) since it was written. The basic steps for setting up a Clojure GAE app haven't changed much, but the Compojure bits may no longer be so simple. You can always just AOT-compile a Clojure HttpServlet to get a web app running.

The release of Java support for Google AppEngine means more than just Java: it means lots of cool JVM languages! My personal favorite is Clojure, so when I and several colleagues got an opportunity to try out a pre-release version (thanks to a partnership between Google and ThoughtWorks) I immediately started trying it out. It turns out to be pretty easy and pretty great.

I'm going to walk you through using Clojure on Google AppEngine by stepping you through the creation of a Clojure/Compojure version of the Guestbook application from the Getting Started section of the AppEngine docs. As we go I'll introduce some bits of appengine-clj, a library I extracted along the way to keep anything that felt like boiler-plate code out of my app.

First you'll need to sign up (if you haven't already) and download the SDK.

Creating Your Application

To start, create a project directory and the basic project directory structure documented in the getting started section of Google's documentation. Here's an overview of what to create and what will go where.

project-root/
  src/
    [Clojure source code]
  war/
    [static files]
    WEB-INF/
      [config files]
      classes/
        [compiled classes from ../../src]
      lib/
        [jar dependencies]

Copy clojure.jar, clojure-contrib.jar, and compojure.jar into WEB-INF/lib. You'll also need to add appengine-api-XXX.jar from the SDK. I'm using fairly recent trunk versions of Clojure, clojure-contrib, and Compojure. (The easiest way to make sure you have compatible versions of these is to pull the latest Compojure source from github and then download their deps.zip and build Compojure against that.) If you want to use appengine-clj you can build it yourself from source or grab a prebuilt jar from the downloads section on Github.

Hello, World!

The entry point to your application will be a servlet class. Create a Clojure source file in your src directory and include a :gen-class directive to extend HttpServlet. Use Compojure's defroutes and defservice to create a HelloWorld.

src/guestbook/servlet.clj
(ns guestbook.servlet
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.http compojure.html))

(defroutes guestbook-app
  (GET "/"
    (html [:h1 "Hello, World!"])))

(defservice guestbook-app)

defroutes above creates a routing function that will respond to an HTML GET for the path "/", and the html function converts vectors to a string of HTML. If you're not familiar with Compojure, start here on the Compojure wiki.

Next create a web.xml with a servlet-mapping sending /* to your servlet class. Since Compojure handles URL routing, your application will have just this one mapping.

war/WEB-INF/web.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app 
    xmlns="http://java.sun.com/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">
  <display-name>Clojure Guestbook</display-name>
  <servlet>
    <servlet-name>gb</servlet-name>
    <servlet-class>guestbook.servlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>gb</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

Create an appengine-web.xml, putting your application ID into the application element.

war/WEB-INF/appengine-web.xml
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>[your application ID]</application>
  <version>1</version>
  <static-files />
  <resource-files />
</appengine-web-app>

Create an ant build.xml file that compiles your application to the classes directory.

build.xml
<project name="guestbook-clj" basedir="." default="compile">
  ... here you'll need to define project.classpath ...
  ... see GitHub for the full working example file ...
  <target name="compile" depends="...">
    <java classname="clojure.lang.Compile" classpathref="project.classpath" failonerror="true">
      <classpath path="${src.dir}" />
      <sysproperty key="clojure.compile.path" value="${classes.dir}" />
      <arg value="guestbook.servlet" />
    </java>
  </target>
</project>

I've left out some of the details, but you can find a full working version on Github.

At this point you should be able to run your "Hello, World" iteration of the application locally using the development appserver and deploy to appspot using the appcfg executable (or an Ant task if you prefer). Now it's just a matter of building your application using the tools Google AppEngine, Clojure, and Compojure make available to you.

The User Service

AppEngine has a simple API for dealing with user accounts. Let's greet logged in users by name. You'll need to import com.google.appengine.api.users.UserServiceFactory.

(ns ...
  (:import
    (com.google.appengine.api.users UserServiceFactory)))

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html [:h1 "Hello, " (if user (.getNickname user) "World") "!"]))))

But we have to let users log in to see this work. The UserService also exposes methods for creating login and logout URLs.

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL user-service "/") "sign in")]
        [:p (link-to (.createLogoutURL user-service "/") "sign out")]))))

Now you should be able to log into your application and be greeted by name. On the dev appserver, the login page will let you provide any username and check a box to indicate whether you should be logged in as an administrator for your application. On the appspot servers, you'll get a proper-looking Google Accounts login page. The argument to createLoginURL and createLogoutURL is the path or URL the user should be redirected to after logging in or out.

I've extracted the basic user-lookup calls into a Ring middleware function and put it into the appengine-clj.users namespace in appengine-clj. Here's what our servlet looks like using that.
(ns guestbook.servlet
  ... you no longer need to import UserServiceFactory ...
  (:require
    [appengine-clj.users :as users]))

(defroutes guestbook-app
  (GET "/"
    (let [user-info (request :appengine-clj/user-info)
          user (user-info :user)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL (user-info :user-service) "/") "sign in")]
        [:p (link-to (.createLogoutURL (user-info :user-service) "/") "sign out")]))))

(defservice (users/wrap-with-user-info guestbook-app))

It's about the same amount of code, it just looks a little more clojurey now. (I'm sorry: that's not a word.)

Datastore

Next let's collect guestbook entries and put them in the Datastore. AppEngine for Java has support for a couple of standard Java persistence APIs, JDO and JPA. But we'll use the lower-level datastore API, which seems a better fit for a dynamic language like Clojure (not to mention it doesn't require us to implement Java classes to persist).

The Java API for datastore is pretty simple, but conceptually it's different enough from a SQL database that it definitely takes some getting used to. (I for one am still figuring it out.)

Here's the sixty-second rundown of the bare essentials. The basic unit of persistence is the Entity, which has a Map of String-keyed properties. An Entity has a kind, which is a string denoting the type. (But keep in mind there's no schema here, so you can give any entity any properties.) An Entity is identified by a Key which is something more than a normal DBMS identifier because it can hold an Entity's association with a parent Entity. Besides using the Key, you can retrieve Entities with a Query, which searches either a single kind of entity or descendent entities of a single ancestor (or both, depending on which constructor you use) and can apply simple filtering and sorting.

The natural Clojurized form of an Entity seemed to be a map, so what I've started pulling out into appengine-clj.datastore is functions that allow Clojure code to work with an immutable map of keyword-keyed properties (plus :key and :kind) and have the library take care of translating into Entity objects. Currently there are just create and find methods, since that was all the basic guestbook needed. (But you know that Internet. I'll need a delete function before the week is out.)

Using appengine-clj.datastore, functions for creating and retrieving guestbook greetings are extremely simple.

(ns guestbook.greetings
  (:require [appengine-clj.datastore :as ds])
  (:import (com.google.appengine.api.datastore Query)))

(defn create [content author]
  (ds/create {:kind "Greeting" :author author :content content :date (java.util.Date.)}))

(defn find-all []
  (ds/find-all (doto (Query. "Greeting") (.addSort "date"))))

Note the creation of a com.google.appengine.api.datastore.Query (in find-all) that pulls back all Greetings and orders them by date. I've considered a couple of approaches for cleaning up creation of a query from Clojure, but I haven't decided between something that looks fairly idiomatic vs something that reads just like GQL. For the time being I'm sticking with the Java-interop style since even that is nicely terse and readable. Take a look at the tests for appengine-clj.datastore for more examples.

Speaking of which, this is a good time to mention that writing tests for datastore code is easy with appengine-clj.test-utils, which provides functions to set up an in-memory datastore. The dstest macro used there creates a fresh datastore for each test. If you're using a different testing framework or prefer different scoping, you can call ds-setup and ds-teardown yourself. (Do keep in mind that this is the development version of datastore, so we'll all need to keep an eye out for differences between that and the real Datastore service.)

HTML and Form Handling

Now that we've got our persistence straight (and tested), let's create a UI so users can sign the guestbook. At this point it's just plain Compojure code. We'll create one route to show the guestbook and a form to enter a greeting at "/" and another route for saving the greeting with a POST to "/sign".

Here's our function for signing the guestbook.

(defn sign-guestbook [params user]
  (greetings/create (params :content) (if user (.getNickname user)))
  (redirect-to "/"))

You can see there's very little to it. It takes the request parameters and a user, calls our greetings/create function, and redirects back to the guestbook.

The function for showing the guestbook is quite a bit more to swallow, since it includes our entire user interface.

(defn show-guestbook [{:keys [user user-service]}]
  (let [all-greetings (greetings/find-all)]
    (html [:html [:head [:title "Guestbook"]]
      [:body
        (if user
          [:p "Hello, " (.getNickname user) "! (You can "
            (link-to (.createLogoutURL user-service "/") "sign out")
            ".)"]
          [:p "Hello! (You can "
            (link-to (.createLoginURL user-service "/") "sign in")
            " to include your name with your greeting when you post.)"])
        (if (empty? all-greetings)
          [:p "The guestbook has no messages."]
          (map (fn [greeting]
            [:div
              [:p (if (greeting :author) [:strong (greeting :author)] "An anonymous guest") " wrote:"]
              [:blockquote (h (greeting :content))]])
            all-greetings))
        (form-to [POST "/sign"]
          [:div (text-area "content" "")]
          [:div (submit-button "Post Greeting")])]])))

It takes the user-info map, which it destructures to grab the user and UserService. It calls our greetings/find-all function to get the items to show and then uses Compojure's html helpers to create the document. For any real application you'd want to break the view down into smaller pieces to avoid such a huge nested chunk of vectors (or consider using another templating library like Enlive), but for this example I think it's easier to understand what's going on with the whole page in one function.

Finally here are the routes that wire it all together.

(defroutes guestbook-app
  (POST "/sign"
    (sign-guestbook params ((request :appengine-clj/user-info) :user)))
  (GET "/"
    (show-guestbook (request :appengine-clj/user-info))))

Here I'm using the :appengine-clj/user-info map that's been assoc'd to the request by the Ring middleware.

See the entire servlet file on GitHub, including some enhancements for styling and to see some other code to exercise Clojure features on AppEngine.

The Big Caveat

Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure.

First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures.

Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache.

More to Come

  • I'll try and expand on this in the future with more write-ups, including a discussion of special handling for static files (which as of the version of the SDK I'm using works great on the appspot servers even with a /* servlet mapping but not on the local dev appserver, where servlet mappings win out over static files).
  • If you'll sign my silly little guestbook on the appspot servers, I'd like to publish information on how many requests I got and how they performed.
  • Google also provides Java APIs for caching, image manipulation, making HTTP requests, and email. I haven't even scratched the surface of those yet.
  • With fresh support in AppEngine for scheduled tasks and upcoming support for task queues, there's more Clojure fun to be had.

Enjoy!

Update 7 September 2009: Late last week, the App Engine Team released version 1.2.5 of the SDK, including both XMPP (jabber instant messaging) support, which is brand new, and Task Queues, which had been available in the Python SDK but are now available for Java (and Clojure) applications.

26 comments:

Unknown said...

Great post!

In order to get this to work with the latest version of compojure, I had to comment out the calls to compojure.http.multipart in request.clj.

There was a security issue with org.apache.commons.fileupload.disk.DiskFileItem referencing java.rmi.server.UID, which is apparently a restricted class.

drcode said...

Thanks John for this post- Very helpful!

...and thanks Zachary for the hint about the multipart error- big time saver.

Anonymous said...

i'm getting an error trying to build the latest from git hub:

$ ant
Buildfile: build.xml

clean:

init:
[mkdir] Created dir: /Users/dude/dev/appengine/guestbook-clj/war/WEB-INF/classes

copyjars:

compile:
[java] Compiling guestbook.servlet to war/WEB-INF/classes

BUILD FAILED
/Users/dude/dev/appengine/guestbook-clj/build.xml:38: java.lang.ExceptionInInitializerError (servlet.clj:1)

Karl Rosaen said...

nice post! a couple of adjustments I needed to make for this to work with the latest in compojure:

1) adjust the use of form-to in servlet.clj since it has changed. see:
http://groups.google.com/group/compojure/browse_thread/thread/e6de5fd77a09ec09#

2) add more jars in the copyjars rule in build.xml, probably because compojure depends on more stuff now. eg:

files="clojure.jar clojure-contrib.jar commons-io-1.4.jar commons-codec-1.3.jar"

basically, you want to copy all the jars from the deps directory, would probably be best to use a glob rule of some sort, but I was too lazy to figure that out in ant

John Hume said...

I've updated guestbook-clj on GitHub to reflect these latest changes to compojure.

Karl, I actually want exactly the two jars you mentioned rather than all the deps, because other jars in deps are either test-only dependencies (fact, re-rand) or inappropriate for an appengine deployment (grizzly, jetty, commons-fileupload, and the servlet API).

Karl Rosaen said...

ah, I see, but commons-io-1.4.jar and commons-codec-1.3.jar were still necessary additions for the latest compojure. thanks updating the project!

Victor Rodriguez said...

Hello,

There is a bug in wrap-requiring-login: it doesn't pass a destination URL to createLoginURL().

Here is my fix (plus an extra functions).

Cheers,

Victor Rodriguez.

diff --git a/src/appengine_clj/users.clj b/src/appengine_clj/users.clj
index 61df7e5..53dc835 100644
--- a/src/appengine_clj/users.clj
+++ b/src/appengine_clj/users.clj
@@ -21,9 +21,14 @@
(application (assoc request :appengine-clj/user-info (user-info)))))

(defn wrap-requiring-login
- [application]
- (fn [request]
- (let [{:keys [user-service]} (:appengine-clj/user-info request)]
- (if (.isUserLoggedIn user-service)
- (application request)
- {:status 302 :headers {"Location" (.createLoginURL user-service)}}))))
+ ([application] (wrap-requiring-login application "/"))
+ ([application destination-url]
+ (fn [request]
+ (let [{:keys [user-service]} (:appengine-clj/user-info request)]
+ (if (.isUserLoggedIn user-service)
+ (application request)
+ {:status 302 :headers {"Location" (.createLoginURL user-service destination-url)}})))))
+
+(defn logout-url
+ ([] (logout-url "/"))
+ ([destination-url] (.createLogoutURL (:user-service (user-info)) destination-url)))

Robin Brandt said...

Did anyone succeed in getting this to work in the 1.2.2 release of the app engine java library?

The unittests in appengine-clj fail and I can't get any results from any Queries. If I try to get an entity by key everything works fine.

Robin Brandt said...

Ok, implementing some more methods in test_utils fixed the problem.

diff --git a/src/appengine_clj/test_utils.clj b/src/appengine_clj/test_utils.clj
index 0c6a4e1..12085c4 100644
--- a/src/appengine_clj/test_utils.clj
+++ b/src/appengine_clj/test_utils.clj
@@ -1,6 +1,7 @@
(ns appengine-clj.test-utils
(:require [clojure.contrib.test-is :as test-is])
(:import
+ [java.util HashMap]
(com.google.appengine.tools.development ApiProxyLocalFactory)
(com.google.appengine.api.datastore.dev LocalDatastoreService)
(com.google.apphosting.api ApiProxy)))
@@ -13,6 +14,8 @@
(ApiProxy/setEnvironmentForCurrentThread
(proxy [com.google.apphosting.api.ApiProxy$Environment] []
(getAppId [] "test")
+ (getRequestNamespace [] "")
+ (getAttributes [] (HashMap. ))
(getVersionId [] "1.0"))))

(defn ds-teardown []

John Hume said...

Robin, I've applied an equivalent change and pushed to github. Let me know if you see any issue. Thanks.

Anonymous said...

If you are getting the error /Users/dude/dev/appengine/guestbook-clj/build.xml:38: java.lang.ExceptionInInitializerError (servlet.clj:1)

and then you turn on debug info for your ant invocation like so:

ant -v -debug

Then you'll see a ClassNotFound

at clojure.lang.Compiler$InvokeExpr.eval(Compiler.java:2714)
... 38 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.fileupload.FileUpload
at org.apache.tools.ant.AntClassLoader.findClassInComponents(AntClassLoader.java:1383)
at org.apache.tools.ant.AntClassLoader.findClass(AntClassLoader.java:1324)

Which is caused by the fact that commons-fileupload-1.2.1.jar is not being put on the classpath in the project.classpath section of the ant build.xml script. I hacked it to add this file and bam, it works.

I'd submit a patch, but want feedback first, because comments by the author above say that the commons-fileupload jar is not desired, but yet it is required for compilation.

John Hume said...

Matthew,
I haven't been tracking Compojure changes closely, but my guess is that it's changed to now depend on fileupload to compile.

Your patch would be much appreciated. Thanks.

Anonymous said...

Being new to Clojure, I think I need a pointer or two. I first added the commons-fileupload JAR and that worked for local dev deployments. However, I actually had to comment out the multipart in Compojure to get it to work on the real AppEngine, since Compojure seems to load those "banned" classes even if you don't use them. So for now, having to use a customized version of Compojure to get this to work on GAE. How does one stop a Clojure project like Compojure from loading classes that are actually never used?

Thanks again for great materials and I hope to contribute to make them even more polished.

John Hume said...

Matthew,
There aren't any Clojure tricks I know of to work around the fact that Compojure loads those libraries eagerly. A change in Compojure's design could give your application control over which parts of the framework get loaded, but that's somewhat at odds with the goal of making it simple to get started with Compojure. I haven't looked into this though, so there may be a way to serve both goals. You could try the Compojure mailing list (where James Reeves is quite responsive) to see what people's thoughts are on a change like this.

Worst case we can create a GAE-compatible fork of Compojure, but I don't have much bandwidth to maintain that, so hopefully we can avoid it.

John Hume said...

FYI, Victor Rodriguez's reported issue with wrap-requiring-login has been fixed. Better (very) late than never.

You can specify a post-login destination URI as in Victor's patch, but if you do a (wrap-requiring-login app) with no URI, it will default to the URI from the original request instead of "/". This ought to provide the desired functionality for the majority of applications.

Robin Brandt said...

Matthew,

we had the same problem and it could be resolved by not using the "include-everything" compojure-module in our clojure-files but all the submodules needed explicitly (e.g. compojure.http.helpers, compojure.http.routes).

This might be a bit of work but it also gives you a better overview of what libraries you use in a file.

Btw., John, what do you think of creating a mailing-list or google group for further discussions of clojure and app engine?

Jeff said...

Any ideas how we could set things up to dynamically reload modifed clojure files from the src directory, rather than having to recompile and run the devserver after every modification?

Robin Brandt said...
This comment has been removed by the author.
Robin Brandt said...

I wrote a small tutorial on our blog about how to develop incrementally without having to recompile everything after a change:

http://hackers-with-attitude.blogspot.com/2009/08/intertactive-programming-with-clojure.html

Michael said...

Thanks for the great tutorial!

I had a couple of issues to get up-and-running.

1) appengine build would not generate jar file; had to do it manually

2) got same problem mentioned by Zachary. I had to grep the compojure directory for "multipart" and comment out 'compojure.http.multipart in compojure.clj and compojure/http.clj, then rebuild compojure.

Then it worked. :-)

Michael said...

On point #2 above -- this appears to be a PC-only problem. I did not have to do anything to get it working on my Mac.

Anonymous said...

The guestbook itself is getting horribly spammy; might want to clear that out if possible.

Thanks for a superb post.

Giacecco said...

Hi! Compojure has changed a lot since you wrote your tutorial. Please put some warning at the beginning of your text, otherwise others may - like me - spend hours trying to make the examples work without realising that many compojure classes have been renamed! Thanks!

François said...

Hi,

Thanks for the nice tutorial! I have a question: how do you debug a Clojure app running on the development server? I am seriously considering building a web cloud app in Clojure. I am hesitating between Clojure and GWT with the full Eclipse IDE experience (debugger, break points, etc...)

John Hume said...

François, have you looked at Counterclockwise? http://code.google.com/p/counterclockwise/wiki/Documentation#Debug_clojure_code

François said...

Hi John,

I played a bit with Counterclockwise. It's very usable. I guess I forgot about it. I was just curious to know how people using Emacs actually debug their code. But you are right: Counterclockwise is probably the best option for me at the moment.

Thanks a lot!