Why Not To Use My Library clj-record

In 2008 I discovered Clojure, and I was extremely excited to finally have an easy entry-way to Lisp. Pretty early on I wanted to get my head around one of Lisp's killer features: macros. A little playing around in the REPL was easy but not satisfying: I wanted to build something powerful.

Having spent a few years doing Ruby on Rails, I thought ActiveRecord's macro-like capabilities (has_many and whatnot) would be a good goal. So I built clj-record.

In retrospect, I went too far trying to build in Rails-like magic, so I want to point out the design flaws that I recommend you avoid in your own work and recommend that you choose something other than clj-record for your Clojure projects that use an RDBMS.

The API of clj-record is all in one macro: clj-record.core/init-model. It commits a few macro sins.

• It looks at the namespace from which you call it to find the name of your model.
• It intentionally captures a db var from your namespace.
• It defs a bunch of fns in your namespace (most of which are just partially applied versions of clj-record.core fns).

So you do this:

and that turns into something like this (leaving out the ns):

This provided really good practice (and an example I've gone back to more than once) for quoting, syntax quoting, unquoting, and unquote-splicing.

But this sucks!

What init-model should do is return a data-structure that you then pass to clj-record's find, insert, and update fns. In addition to being simpler to understand than the hidden registry of model metadata that clj-record has to maintain, it would make it more natural to write generic, composable code, since the model itself would be a value.

If that were all init-model needed to do, the code (both inside clj-record and user code) would probably be even simpler if there were no macro at all. A set of functions could enhance the model data-structure as needed so that even model setup code could be composed like any other series of fn-calls.

Maybe it would be worthwhile having a macro that went something like this:

This macro would do a def, but it's completely intuitive what that def is. The third "sin" on my list above was about defining a bunch of fns in your namespace. I call this a sin because it's not clear from reading the code that there are a bunch of vars defined, making both discovery and debugging more difficult. There may be cases where "hidden defs" are justified by performance gains or code reduction, but the fns defined by init-model don't pass that test (because (clj-record.core/find-records widget attributes) is no worse and more easily composed with other generic model-processing fns than (my.model.widget/find-records attributes)).

If no one else were writing libraries, I'd try to remedy this in the next major revision of clj-record, but someone else has done this work (or something very much like it) already. Chris Granger wrote korma, and it offers similar functionality in a saner way. I'm also not using Clojure with a SQL database, so I wouldn't be able to eat my own dogfood.

If you're modeling entities in a relational database, I'd recommend korma over clj-record. There may be even better options for you (including just using clojure.java.jdbc directly), so look around.

If you're writing macros, don't capture vars, don't make their functionality depend on globals like *ns*, and consider alternatives to creating vars in the caller's namespace. While you're at it, consider whether the job would be better done with plain old functions.

Using Aleph's Asynchronous HTTP Client for OAuth

If you're implementing authentication in your web app on the server-side using some third party's OAuth, you ideally want to dedicate minimal system resources to waiting for HTTP responses. If you're working in Clojure, you can do this using Aleph's asynchronous http client. Here I'll walk through using it to do server-side authentication with Facebook.

[There's nothing auth-specific about this. Any integration with HTTP end points might benefit from similar treatment.]

Here's what our application needs to do:

1. Send the user to Facebook's OAuth "dialog," passing along a redirect_uri parameter. On the dialog they'll see what permissions your app wants and click a button to say they want to log in. Facebook will redirect them back to your redirect_uri with a code parameter in the query string.
2. Make a server-side request for an access-token for the user, passing along the code parameter and your own redirect_uri
3. Make a second server-side request for whatever information the app needs (e.g., the user's name and email address), passing along the access-token.

We can't do anything for the user between steps two and three, so we'll perform them one after another. It's easy to picture what that might look like if we aren't worried about tying up a thread waiting for responses.

Ideally we'd like an asynchronous implementation to read just as clearly.

We'll start with the easiest implementation to understand, which uses on-realized to register success and error callbacks on each HTTP request. The fn now takes success and error callbacks. In an aleph web application, those would each enqueue a ring-style response map onto the response channel.

Unfortunately this reads terribly due to the nested callbacks.

We can use lamina's run-pipeline macro to create a cleaner version of the same thing.

At this point, however, we realize that we're missing something. In addition to the user data we get from the second HTTP request, we want the "fb-user" passed to the success callback to have the access-token and access-token-expiration we got in response to the first HTTP request.

We can achieve that by breaking the pipeline into two and holding onto the result of the first so that we can refer to it again later (in the call to combine-results.

Note the deref ("@") in that second reference to access-token-result. It's needed because the pipeline returns a result-channel which may not yet be realized. We don't have to worry about the deref blocking, since the value is guaranteed to be realized before the second pipeline will proceed past the initial value, but it looks like something that might block. The whole thing's also a bit sloppy. Zack Tellman pointed out that it's a bit cleaner to split them like so.

The deref is no longer needed because the nested fn will be called with the realized value once it's ready.

If we extract the details so that we have the same level of abstraction we had in the initial (synchronous) version, we end up with this.

Comparing that last fn to the synchronous version, we come out looking pretty good here! The only thing I find a little awkward is that the options to run-pipeline are the second argument. It breaks up the flow when the first "value" is just a call to another pipeline.

Assorted Thoughts

Reusing the connection

Aleph's http-client fn lets you reuse one connection for multiple requests to the same host. It mucks things up a bit, since you want to be sure the connection gets closed at the end.

Not being able to just wrap a (try ... (finally ...)) around the whole thing is a bummer, but it's still not awful. Zach has said there will likely soon be a cleaner way to do "finally" in pipelines.

The :error-handler

The empty :error-handler fns on the "get-fb-" fns suppress "unhandled exception" logging from lamina. Since both of those pipelines are run as part of the outer pipeline, when either of them errors, the error-handler from the outer pipeline will be run, so that's the only place we need to use the error-callback. In my real app, those inner error-handlers just log, but you could just as easily leave the options out completely if you're ok with lamina's logging.

The rest

I've left out many details, like the FB-specific URL-templates and response parsing. If there's interest, I'm happy to share those. I just thought the lamina/aleph stuff was the interesting part.

How to Handle File Uploads from Clojure on Heroku (or anywhere else)

I recently wanted to deploy a Clojure web app that allowed users to upload a CSV and work with the data. Heroku is as easy as it gets for web app deployment, but I figured I'd have to jump through some extra hoops to do the file upload. It turns out there are no hoops. Thanks to Heroku's ephemeral filesystem, we can handle file uploads just as we would on any other system, so long as we either process the file immediately or send it someplace else to be stored.

Here's how to handle a file upload in a very simple ring app.

The 'render-' fns are left to you!

FluentTime: a fluent interface in C# for working with dates and times

After working fairly late last night I was too wound up to go to sleep when I got home. Something possessed me to start creating a little C# library providing a fluent interface for working with DateTimes, DateTimeOffsets, and TimeSpans inspired by ActiveSupport's active_support/core_ext/numeric/time and related extensions.

After about 3.Hours(30.Minutes()) the straightforward part had come together. You can do stuff like this with it:

var toothBrushingTime = 10.Minutes().Before(bedTime);
var lightsOutForReal = 1.Hour(30.Minutes()).After(bedTime);
...
Assert.That(actual, Is.LessThan(50.Milliseconds().After(expected)));

These built in .NET types are pretty easy to work with and made the implementation very easy (except for the bit where I discovered that TimeSpan.FromMilliseconds takes a double but only has whole-millisecond precision).

This morning I put it up on Github. You can find FluentTime here.

The one more complex thing I'd still like to add is a variable-length variation on TimeSpan for handling units like months and years. I'm interested to see how clean I can keep things when my type will only take on a specific value based on the date-time value it's being added to or subtracted from. That ought to be a good opportunity for exploring operator overloading. (Unsurprisingly that hasn't come up in my day job just yet.)

My Favorite Analysis Artifact: Code

On projects where you're integrating with legacy or vendor systems, it's often necessary to dedicate developers to technical analysis of the system before stories can be broken up reasonably and estimated by the team. Typically the output from that analysis is technical notes on a wiki page (or Mingle card). Those notes are helpful, but keep an eye out for the opportunity to do something far more useful: automated specs showing how to get the system to do what you need (or something close, if the details haven't been worked out).

Since your specs are self-verifying, whoever picks up the story will know for sure that the system works the way the specs say, which is never the case for notes on a wiki.

Code is a natural and unambiguous way to describe using a system, so it eliminates challenges of interpreting someone else's notes. It's also likely faster for developers to write code than to write prose.

Resist the temptation to go beyond the spec and write production implementation code. If your requirements haven't been fully baked, you'll just be bloating the codebase. Your specs can be clearly labeled as analysis artifacts, and once the real integration has happened, they can be deleted.

I'd recommend using this approach any time you already have a test environment for the legacy database or application. Besides giving you more manageable and useful analysis artifacts, it's likely to keep the developers assigned to do technical analysis much happier.

ActionScript const Gotcha

I've been working in ActionScript lately. It's a bizarre language. Java-style OO features are smashed right on top of JavaScript, with no regard for the prototype-based inheritance that the language has had all along. (As far as I can tell Adobe doesn't want you to think about prototypes. I haven't dug into how the Java-style ActionScript class model plays with the prototype model, but it might be worth exploring.)

After a few years of full-time Ruby, the adjustment has been pretty painful. One of my biggest complaints is the verbosity of the static type declarations. Having recently spent some time playing around with Scala, I was incredibly impressed at how much noise can be removed from a statically typed language with a compiler that does good type-inference. The ActionScript compiler does absolutely zero type-inference, and the result makes for really nasty reading.

One of the additions to JavaScript syntax that initially impressed me was the introduction of the const keyword. I really appreciate how Scala's val keyword makes declaration of immutable fields just as terse and clear as the var keyword does for mutable fields. This is in contrast to Java, where making things immutable requires an extra keyword (final) that makes code with immutables (i.e., better, safer code) extra noisy, a great disincentive to doing the right thing. Using const in place of var in ActionScript is much prettier than the Java way. Unfortunately I quickly discovered a couple of shortcomings in ActionScript "constants" that really bummed me out.

The first issue I ran into very quickly: Whereas Java only allows final variables to be assigned once, it's smart enough to allow that assignment to happen in a constructor. The ActionScript compiler specifically requires them to be assigned at the point of declaration. This means you can't declare a constant field and assign it its value in a constructor, which is the primary reason I'd want constant fields.

Boo! But const was still good for static constants (obviously) and seemed to be good for locals as well.

The second issue has to do with using const on locals and led to a pretty painful debugging session. We were taking advantage of one feature ActionScript holds high over Java's head: functions are closures, and closures are handy. I was looping over some data structure using a for loop. Inside that loop I declared a const, then declared a function I wanted for later that used that const. But at the end of the loop all the functions were referring to the last item in the data structure.

Here's some sample code illustrating what I was seeing.

// Maps an array to an array of functions that return
// the values from the original array.
// But doesn't work!
function identityFunctions(items:Array):Array {
  const functions:Array = new Array();
  for (var i:int = 0; i < items.length; i++) {
    const item:* = items[i];
    functions[i] = function():* { return item; };
  }
}

const funcs:Array = identityFunctions(["a", "b", "c"]);
funcs[2](); // => "c", as desired
funcs[1](); // => "c", but I wanted "b"
funcs[0](); // => "c", but I wanted "a"

Here's something I either never knew or at some point forgot about JavaScript: variables are lexically scoped, but only function bodies introduce new lexical scopes. So even though that item const is declared inside the loop, it's really scoped to the entire function. It gets assigned a new value on each pass of the loop, so yes, the const keyword is a bold-faced lie. Apparently the keyword tells the compiler not to let any other code assign to that reference, but it creates a mutable reference just like var, and in this case it gets repointed several times.

Basically the function definition above compiles into the same thing as this.

// Comes right out and tells you it doesn't work.
function identityFunctions(items:Array):Array {
  const functions:Array = new Array();
  var item:*;
  for (var i:int = 0; i < items.length; i++) {
    item = items[i];
    functions[i] = function():* { return item; };
  }
}

Certainly it's my bad not knowing that the for loop doesn't introduce a lexical scope, but it's crazy for the compiler to allow declaration of a const inside a looping structure. If Adobe's really committed to all this static typing and compiler checking hooey, surely they'd agree this is a nasty gotcha that the compiler should prevent.

As far as workarounds, there are several. For the contrived example above, the simplest is to use one of the iteration methods Array has in ActionScript. There the "body" of the loop is a function, so it has its own scope. Here's the cleanest way.

// Actually works.
function identityFunctions(items:Array):Array {
  return items.map(function(item:*, i:int, a:Array):* {
    return function():* { return item; };
  });
}

RapidSMS at Away Day

Last weekend at the ThoughtWorks US Away Day we were joined by Merrick Schaefer from UNICEF Innovation and a couple developers who've worked extensively with RapidSMS. RapidSMS is an open source project providing a set of libraries and apps for interacting with phones. It's been used for a variety of purposes, mostly in Africa, where cell phone coverage is far better than electricity or land-line availability. One deployment they talked through in detail allows (relatively) untrained medical workers to use SMS messages to sumbit a few measurements to assess childhood malnutrition. In addition to recording the information to keep accurate and instantly available statistics, the system performs an immediate diagnosis and texts it back to the worker in the field.

Jeff Wishnie organized a RapidSMS code jam for those not in the mood to attend sessions, and Brian Guthrie and I spent most of Saturday pairing on a simple feature addition to one of the support applications. It was the first time either of us had touched Python for more than a few minutes, and the first time we'd touched Django at all, so it was an interesting learning experience. It was also extremely rewarding knowing that we were contributing to such a good cause.

If you sometimes find yourself in the mood for some coding but don't always have a personal project burning to get out, I'd highly encourage you to spend a little time familiarizing yourself with RapidSMS and maybe find something in their issue tracker you could tackle. I promise you'll feel great about it.

Interactive Clojure-AppEngine Development

If you like (1) interactive programming in Clojure from Emacs and (2) deploying to Google AppEngine, but you haven't yet managed to bring these two together, you should take a look at this post over at hackers-with-attitude, which walks you through setting up a pretty slick workflow.

SimURLator: Test Custom iPhone URL Schemes in the iPhone Simulator

Over the last few months ThoughtWorks has implemented several "Conference Connection" iPhone applications. They provide a view of the conference schedule, the ability to save a list of sessions you plan to attend, maps of the conferene site, a view of relevant Twitter posts, and some features specific to the various conferences.

The Twitter integration includes a feature to post, launching your prefered Twitter client (Tweetie, Echofon—formerly "TwitterFon"—or Twitterrific) using UIApplication's openURL: and the apps' custom URL schemes.

Relatively late in the testing cycle for another upcoming conference, we discovered an issue with selecting a Twitter client when you have more than one of the supported clients installed. Some of our testers were testing only in the iPhone Simulator, and because you can't install third-party apps in the Simulator, that feature had been undertested.

That was no good, so we created a sort of workaround: a stand-in app that takes the place of the third-party apps you can't install in the Simulator. SimURLator is now available on Github.

You configure SimURLator to respond to whatever custom URL schemes you want (by adding or removing URL schemes in SimURLator-Info.plist) and deploy it to your Simulator alongside your app. When SimURLator is launched via a URL by your app, it shows you the full URL. Obviously that's not as satisfying as seeing the third-party app launch and do the right thing, but it's a lot better than nothing.

You can also test your app's handling of its own custom URL schemes. Launch SimURLator and type in a URL that should launch your application. One button lets you verify that the OS reports the URL can be opened (with canOpenURL:); the other launches your app (with openURL:).

If there are other features you think would be useful, comment below. Better yet, fork it and show us.

Google Group for clj-record

The title pretty much says it all. There's now a Google Group for discussion of clj-record: clj-record-dev.

Renum 1.1: pretty instance-specific method definition

You could always define instance-specific behavior on Renum values the same way you can on any object in Ruby:

enum :FakeWord do
  Foo()
  Bar()
  Baz()
  
  def extra_stuff
    "no extra stuff here"
  end
end

def Foo.extra_stuff
  "here's the foo stuff"
end

That works fine, but it's a little ugly.

Starting this afternoon (with release 1.1) you can do this instead:

enum :FakeWord do
  Foo do
    def extra_stuff
      "here's the foo stuff"
    end
  end
  Bar()
  Baz()
  
  def extra_stuff
    "no extra stuff here"
  end
end

FakeWord::Foo will have its own implementation of extra_stuff while FakeWord::Bar and FakeWord::Baz will have the one defined inside the enum block. This was implemented by just instance_eval'ing the given block against the instance just after creating it. A def form in an instance_eval'd block ends up defining a singleton method, so it's equivalent to the uglier version above.

This ends up looking almost exactly like the Java enum equivalent for defining constant-specific class bodies.

Since the block is instance_eval'd, if you prefered you could do any initialization there instead of defining an init method. Depending on what you've got going on it may turn out to be more readable. Here's a contrived example from the spec translated to that style.

# init-with-args-style
enum :Size do
  Small("Really really tiny")
  Medium("Sort of in the middle")
  Large("Quite big")

  attr_reader :description

  def init description
    @description = description
  end
end

# instance-block-per-value-style
enum :Size do
  Small  { @description = "Really really tiny" }
  Medium { @description = "Sort of in the middle" }
  Large  { @description = "Quite big" }
  
  attr_reader :description
end

The latter is probably slightly easier to digest, but it's also an extremely simple case, so I'd expect your mileage to vary quite a bit. (Note that you can do both an init with arguments AND an associated block. Renum evals the block before calling init, so any instance-specific behavior is in place before your init runs.)

Anyway, comment, suggest, complain, fork, improve, and enjoy.

AR⊗TA

If you haven't already, you should check out Brian Marick's Agile reboot: Artisanal Retro-Futurism Crossed With Team Scale Anarcho-Syndicalism.

There's a video of his excellent presentation from Agile Roots if you don't feel like reading. There are also stickers you can ask for if you feel like spreading the word.

It's totally tangential to the point, but my favorite quote from the video is near the end of the questions: "...in my observation agile teams care more about delivering business value, often, than the business they're delivering it to."

A Quick Look at Azure, Microsoft's Cloud

Last week at a ThoughtWorks home-office gathering in New York we had a presentation by one of Microsoft's Architecture Evangelists about their in-the-works cloud offering, Azure. I have to say I was pretty impressed. (If not for the whole Windows thing, I'd probably be trying it out instead of writing about it right now. If you aren't as bothered by Windows as I am, you can register and try it yourself.) Here are some fairly random impressions, mostly based on comparison to the two other best-known cloud contenders.

Azure conceptually leans more towards Google AppEngine's "scale me up; scale me down; hide the details" approach than Amazon EC2's easy-to-manage-infrastructure-as-a-service model, though it currently sits somewhere between them. There's not (yet, at least) any automatic scaling up and down, which surprised me. However what you manually configure is just the number of instances of each application, which is far simpler than manually spinning up new instances of EC2 images and getting them into your cluster (or what-have-you). (Although if you've set up your infrastructure right, your EC2 scaling activities could be just as simple. EC2 just gives you the freedom to do it badly.)

There are two-and-a-half types of application: web apps, worker apps, and web+worker apps. The worker concept is very cool and not something you can do with AppEngine. (The closest thing is their support for scheduled tasks, but these are limited to scheduled URL hits no more often than once a minute with the same restrictions as a normal web request.) Apps all run on something pretty close to the normal .NET framework (minus normal file system access and presumably some other areas--I forgot to ask about threading.)

There are three kinds of storage.

• A file-system-like BLOB storage API, which involves saving individual blocks and then telling the system in what order to stitch them together. It seemed clunky, but I could see writing a simple abstraction over it for small stuff (where you'd usually only need one block) and for large stuff maybe being able to take advantage of already having some blocks uploaded when something gets interrupted and can resume later.
• A BigTable-like "table" storage API. It isn't as rich as AppEngine's BigTable-backed Datastore in that entity properties are restricted to very simple types (no lists, for example) and there's no built-in support for relationships of any kind (even the simple hierarchy of Datastore). (Note there's nothing to stop you from storing keys of other entities as property values and rolling your own awkward relationships.) Similar to Datastore on AppEngine, you have to think about how you want your data partitioned and make some under-informed guesses about slow queries across multiple partitions vs parallel queries each in their own partition. (On Azure, the key for each entity includes an app-generated token naming the partition in which it should reside, which is a bit simpler than Datastore's parentage-based entity groups.) The story on transactions (being limited to a single entity group) seems to be similar as well. There's also currently no indexing by anything other than key, though it sounded like they'll allow you to add at least one additional index per table before they go proper-live.
• Most interestingly, damn-near plain old SQL Server storage in the form of SQL Data Services, with just about every feature of MS SQL Server any normal application would use (except distributed transactions). I believe this is a separate service that you'll need to sign up for and eventually pay for separately. Regardless, it's a key element of the offering, because it provides a huge boost to the suggestion that you can migrate existing .NET applications into the cloud.

There are also persistent queues, which web apps can use to communicate to worker apps asynchronously. (I guess this qualifies as a fourth kind of storage.) They feature guaranteed delivery (with possible repeated delivery) of messages. This is very cool and not something AppEngine offers. (Even if you rolled your own queue-ish thing in datastore, there's no facility to consume messages intelligently.)

There are REST APIs (and SOAP as well, I think) for just about everything, so you have a lot of options for architectures (using heterogeneous technologies) that run partly on-premisis and partly in the Azure cloud.

There are a bunch of other services you can hook into, including ".NET Services" (low-level technical stuff including a Service Bus, Access Control, and a Workflow Service) and "Live Services" (higher-level end-user-meaningful services such as Contacts, Windows Live ID, and assorted Bing stuff).

There's also, surprisingly, the ability to run unmanaged code in your .NET application, AND some form of support for non-.NET stuff via IIS's FastCGI support.

Anyway, it looks like it'll be a compelling offering, competitive with EC2-hosted Windows, and thanks to SQL Data Services, much closer to allowing you to put a normal .NET web app into the cloud than AppEngine can do for Java web apps. For any organization willing to trust Microsoft with its infrastructure, Azure is a good reason to be excited. For the rest of the world, carefully executed EC2 still looks like the best bet for the Enterprise in the cloud.

Renum Moved to Github

Chris O'Meara recently contributed a fix to make constantize_attribute handle nil values sanely. (Did you know "".constantize # => Object? I was sure surprised.)

He's using constantize_attribute to persist renum enumeration values in ActiveRecord models, which is the reason I wrote it in the first place. The reminder that these things do get used here and there inspired me to give renum a little late Spring cleaning.

Renum now lives on github and is set up to be gem-packaged with jeweler, which is a lot more elegant than the previous solution (at least in terms of how much extra stuff it puts into source control). Release 1.0.2 is tagged, but it's functionally the same as 1.0.1 (it just lives in a better SCM and has less packaging cruft).

clj-record now has find-by-sql

I don't know why I didn't do this 3 months ago (or 5), but this morning I extracted find-by-sql from find-records so you can do arbitrary queries and still have them run through the callbacks defined for your model.

Specifying Column Order in ActiveRecord Migrations

ActiveRecord migrations don't allow you to control column order when adding or changing columns. You can always just

execute "ALTER TABLE ..." # but it's ugly and hard to remember the details.

If you're willing to give up vendor neutrality, you can have the feature in pretty-Ruby-migration syntax with a small effort. Here's an implementation for MySQL that adds options :first and :after to add_column and change_column.

ActiveRecord::ConnectionAdapters::MysqlAdapter.class_eval do
  def add_column_options!(sql, options)
    super
    if options[:after]
      sql << " AFTER #{quote_column_name(options[:after])}"
    elsif options[:first]
      sql << " FIRST"
    end
  end
end

(It's also available in this gist if you'd like to suggest a cleaner way.)

With that in place,

a.add_column :foos, :bar, :string, :after => :baz

will execute

ALTER TABLE `foos` ADD `bar` varchar(255) AFTER `baz`

and

a.add_column :foos, :bar, :string, :first => true

will execute

ALTER TABLE `foos` ADD `bar` varchar(255) FIRST

Love that Ruby!

Now this is a monkey-patch, but as monkey-patches go it's pretty innocuous. Rather than redefining a method that's already implemented in MysqlAdapter (and as with many monkey-patches copy-pasting the existing implementation so we can modify it), we're defining a new override of a method (albeit undocumented) inherited from the SchemaStatements module (via AbstractAdapter). If you wanted to, you could subclass MysqlAdapter and then configure Rails to use your own adapter, but I suspect that would be more expensive to own than just adding this method to MysqlAdapter. (Beware: there are many useful Rails-provided rake tasks that look at your adapter string to decide whether to be useful or to spit in your face.)

Clojure on Google AppEngine

Note: This post is quite old, and Compojure has changed (and gotten quite a bit bigger) since it was written. The basic steps for setting up a Clojure GAE app haven't changed much, but the Compojure bits may no longer be so simple. You can always just AOT-compile a Clojure HttpServlet to get a web app running.

The release of Java support for Google AppEngine means more than just Java: it means lots of cool JVM languages! My personal favorite is Clojure, so when I and several colleagues got an opportunity to try out a pre-release version (thanks to a partnership between Google and ThoughtWorks) I immediately started trying it out. It turns out to be pretty easy and pretty great.

I'm going to walk you through using Clojure on Google AppEngine by stepping you through the creation of a Clojure/Compojure version of the Guestbook application from the Getting Started section of the AppEngine docs. As we go I'll introduce some bits of appengine-clj, a library I extracted along the way to keep anything that felt like boiler-plate code out of my app.

First you'll need to sign up (if you haven't already) and download the SDK.

Creating Your Application

To start, create a project directory and the basic project directory structure documented in the getting started section of Google's documentation. Here's an overview of what to create and what will go where.

project-root/
  src/
    [Clojure source code]
  war/
    [static files]
    WEB-INF/
      [config files]
      classes/
        [compiled classes from ../../src]
      lib/
        [jar dependencies]

Copy clojure.jar, clojure-contrib.jar, and compojure.jar into WEB-INF/lib. You'll also need to add appengine-api-XXX.jar from the SDK. I'm using fairly recent trunk versions of Clojure, clojure-contrib, and Compojure. (The easiest way to make sure you have compatible versions of these is to pull the latest Compojure source from github and then download their deps.zip and build Compojure against that.) If you want to use appengine-clj you can build it yourself from source or grab a prebuilt jar from the downloads section on Github.

Hello, World!

The entry point to your application will be a servlet class. Create a Clojure source file in your src directory and include a :gen-class directive to extend HttpServlet. Use Compojure's defroutes and defservice to create a HelloWorld.

src/guestbook/servlet.clj

(ns guestbook.servlet
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.http compojure.html))

(defroutes guestbook-app
  (GET "/"
    (html [:h1 "Hello, World!"])))

(defservice guestbook-app)

defroutes above creates a routing function that will respond to an HTML GET for the path "/", and the html function converts vectors to a string of HTML. If you're not familiar with Compojure, start here on the Compojure wiki.

Next create a web.xml with a servlet-mapping sending /* to your servlet class. Since Compojure handles URL routing, your application will have just this one mapping.

war/WEB-INF/web.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app 
    xmlns="http://java.sun.com/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">
  <display-name>Clojure Guestbook</display-name>
  <servlet>
    <servlet-name>gb</servlet-name>
    <servlet-class>guestbook.servlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>gb</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

Create an appengine-web.xml, putting your application ID into the application element.

war/WEB-INF/appengine-web.xml

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>[your application ID]</application>
  <version>1</version>
  <static-files />
  <resource-files />
</appengine-web-app>

Create an ant build.xml file that compiles your application to the classes directory.

build.xml

<project name="guestbook-clj" basedir="." default="compile">
  ... here you'll need to define project.classpath ...
  ... see GitHub for the full working example file ...
  <target name="compile" depends="...">
    <java classname="clojure.lang.Compile" classpathref="project.classpath" failonerror="true">
      <classpath path="${src.dir}" />
      <sysproperty key="clojure.compile.path" value="${classes.dir}" />
      <arg value="guestbook.servlet" />
    </java>
  </target>
</project>

I've left out some of the details, but you can find a full working version on Github.

At this point you should be able to run your "Hello, World" iteration of the application locally using the development appserver and deploy to appspot using the appcfg executable (or an Ant task if you prefer). Now it's just a matter of building your application using the tools Google AppEngine, Clojure, and Compojure make available to you.

The User Service

AppEngine has a simple API for dealing with user accounts. Let's greet logged in users by name. You'll need to import com.google.appengine.api.users.UserServiceFactory.

(ns ...
  (:import
    (com.google.appengine.api.users UserServiceFactory)))

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html [:h1 "Hello, " (if user (.getNickname user) "World") "!"]))))

But we have to let users log in to see this work. The UserService also exposes methods for creating login and logout URLs.

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL user-service "/") "sign in")]
        [:p (link-to (.createLogoutURL user-service "/") "sign out")]))))

Now you should be able to log into your application and be greeted by name. On the dev appserver, the login page will let you provide any username and check a box to indicate whether you should be logged in as an administrator for your application. On the appspot servers, you'll get a proper-looking Google Accounts login page. The argument to createLoginURL and createLogoutURL is the path or URL the user should be redirected to after logging in or out.

I've extracted the basic user-lookup calls into a Ring middleware function and put it into the appengine-clj.users namespace in appengine-clj. Here's what our servlet looks like using that.

(ns guestbook.servlet
  ... you no longer need to import UserServiceFactory ...
  (:require
    [appengine-clj.users :as users]))

(defroutes guestbook-app
  (GET "/"
    (let [user-info (request :appengine-clj/user-info)
          user (user-info :user)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL (user-info :user-service) "/") "sign in")]
        [:p (link-to (.createLogoutURL (user-info :user-service) "/") "sign out")]))))

(defservice (users/wrap-with-user-info guestbook-app))

It's about the same amount of code, it just looks a little more clojurey now. (I'm sorry: that's not a word.)

Datastore

Next let's collect guestbook entries and put them in the Datastore. AppEngine for Java has support for a couple of standard Java persistence APIs, JDO and JPA. But we'll use the lower-level datastore API, which seems a better fit for a dynamic language like Clojure (not to mention it doesn't require us to implement Java classes to persist).

The Java API for datastore is pretty simple, but conceptually it's different enough from a SQL database that it definitely takes some getting used to. (I for one am still figuring it out.)

Here's the sixty-second rundown of the bare essentials. The basic unit of persistence is the Entity, which has a Map of String-keyed properties. An Entity has a kind, which is a string denoting the type. (But keep in mind there's no schema here, so you can give any entity any properties.) An Entity is identified by a Key which is something more than a normal DBMS identifier because it can hold an Entity's association with a parent Entity. Besides using the Key, you can retrieve Entities with a Query, which searches either a single kind of entity or descendent entities of a single ancestor (or both, depending on which constructor you use) and can apply simple filtering and sorting.

The natural Clojurized form of an Entity seemed to be a map, so what I've started pulling out into appengine-clj.datastore is functions that allow Clojure code to work with an immutable map of keyword-keyed properties (plus :key and :kind) and have the library take care of translating into Entity objects. Currently there are just create and find methods, since that was all the basic guestbook needed. (But you know that Internet. I'll need a delete function before the week is out.)

Using appengine-clj.datastore, functions for creating and retrieving guestbook greetings are extremely simple.

(ns guestbook.greetings
  (:require [appengine-clj.datastore :as ds])
  (:import (com.google.appengine.api.datastore Query)))

(defn create [content author]
  (ds/create {:kind "Greeting" :author author :content content :date (java.util.Date.)}))

(defn find-all []
  (ds/find-all (doto (Query. "Greeting") (.addSort "date"))))

Note the creation of a com.google.appengine.api.datastore.Query (in find-all) that pulls back all Greetings and orders them by date. I've considered a couple of approaches for cleaning up creation of a query from Clojure, but I haven't decided between something that looks fairly idiomatic vs something that reads just like GQL. For the time being I'm sticking with the Java-interop style since even that is nicely terse and readable. Take a look at the tests for appengine-clj.datastore for more examples.

Speaking of which, this is a good time to mention that writing tests for datastore code is easy with appengine-clj.test-utils, which provides functions to set up an in-memory datastore. The dstest macro used there creates a fresh datastore for each test. If you're using a different testing framework or prefer different scoping, you can call ds-setup and ds-teardown yourself. (Do keep in mind that this is the development version of datastore, so we'll all need to keep an eye out for differences between that and the real Datastore service.)

HTML and Form Handling

Now that we've got our persistence straight (and tested), let's create a UI so users can sign the guestbook. At this point it's just plain Compojure code. We'll create one route to show the guestbook and a form to enter a greeting at "/" and another route for saving the greeting with a POST to "/sign".

Here's our function for signing the guestbook.

(defn sign-guestbook [params user]
  (greetings/create (params :content) (if user (.getNickname user)))
  (redirect-to "/"))

You can see there's very little to it. It takes the request parameters and a user, calls our greetings/create function, and redirects back to the guestbook.

The function for showing the guestbook is quite a bit more to swallow, since it includes our entire user interface.

(defn show-guestbook [{:keys [user user-service]}]
  (let [all-greetings (greetings/find-all)]
    (html [:html [:head [:title "Guestbook"]]
      [:body
        (if user
          [:p "Hello, " (.getNickname user) "! (You can "
            (link-to (.createLogoutURL user-service "/") "sign out")
            ".)"]
          [:p "Hello! (You can "
            (link-to (.createLoginURL user-service "/") "sign in")
            " to include your name with your greeting when you post.)"])
        (if (empty? all-greetings)
          [:p "The guestbook has no messages."]
          (map (fn [greeting]
            [:div
              [:p (if (greeting :author) [:strong (greeting :author)] "An anonymous guest") " wrote:"]
              [:blockquote (h (greeting :content))]])
            all-greetings))
        (form-to [POST "/sign"]
          [:div (text-area "content" "")]
          [:div (submit-button "Post Greeting")])]])))

It takes the user-info map, which it destructures to grab the user and UserService. It calls our greetings/find-all function to get the items to show and then uses Compojure's html helpers to create the document. For any real application you'd want to break the view down into smaller pieces to avoid such a huge nested chunk of vectors (or consider using another templating library like Enlive), but for this example I think it's easier to understand what's going on with the whole page in one function.

Finally here are the routes that wire it all together.

(defroutes guestbook-app
  (POST "/sign"
    (sign-guestbook params ((request :appengine-clj/user-info) :user)))
  (GET "/"
    (show-guestbook (request :appengine-clj/user-info))))

Here I'm using the :appengine-clj/user-info map that's been assoc'd to the request by the Ring middleware.

See the entire servlet file on GitHub, including some enhancements for styling and to see some other code to exercise Clojure features on AppEngine.

The Big Caveat

Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure.

First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures.

Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache.

More to Come

• I'll try and expand on this in the future with more write-ups, including a discussion of special handling for static files (which as of the version of the SDK I'm using works great on the appspot servers even with a /* servlet mapping but not on the local dev appserver, where servlet mappings win out over static files).
• If you'll sign my silly little guestbook on the appspot servers, I'd like to publish information on how many requests I got and how they performed.
• Google also provides Java APIs for caching, image manipulation, making HTTP requests, and email. I haven't even scratched the surface of those yet.
• With fresh support in AppEngine for scheduled tasks and upcoming support for task queues, there's more Clojure fun to be had.

Enjoy!

Update 7 September 2009: Late last week, the App Engine Team released version 1.2.5 of the SDK, including both XMPP (jabber instant messaging) support, which is brand new, and Task Queues, which had been available in the Python SDK but are now available for Java (and Clojure) applications.