Specifying Column Order in ActiveRecord Migrations

ActiveRecord migrations don't allow you to control column order when adding or changing columns. You can always just

execute "ALTER TABLE ..." # but it's ugly and hard to remember the details.

If you're willing to give up vendor neutrality, you can have the feature in pretty-Ruby-migration syntax with a small effort. Here's an implementation for MySQL that adds options :first and :after to add_column and change_column.

ActiveRecord::ConnectionAdapters::MysqlAdapter.class_eval do
  def add_column_options!(sql, options)
    super
    if options[:after]
      sql << " AFTER #{quote_column_name(options[:after])}"
    elsif options[:first]
      sql << " FIRST"
    end
  end
end

(It's also available in this gist if you'd like to suggest a cleaner way.)

With that in place,

a.add_column :foos, :bar, :string, :after => :baz

will execute

ALTER TABLE `foos` ADD `bar` varchar(255) AFTER `baz`

and

a.add_column :foos, :bar, :string, :first => true

will execute

ALTER TABLE `foos` ADD `bar` varchar(255) FIRST

Love that Ruby!

Now this is a monkey-patch, but as monkey-patches go it's pretty innocuous. Rather than redefining a method that's already implemented in MysqlAdapter (and as with many monkey-patches copy-pasting the existing implementation so we can modify it), we're defining a new override of a method (albeit undocumented) inherited from the SchemaStatements module (via AbstractAdapter). If you wanted to, you could subclass MysqlAdapter and then configure Rails to use your own adapter, but I suspect that would be more expensive to own than just adding this method to MysqlAdapter. (Beware: there are many useful Rails-provided rake tasks that look at your adapter string to decide whether to be useful or to spit in your face.)

Clojure on Google AppEngine

Note: This post is quite old, and Compojure has changed (and gotten quite a bit bigger) since it was written. The basic steps for setting up a Clojure GAE app haven't changed much, but the Compojure bits may no longer be so simple. You can always just AOT-compile a Clojure HttpServlet to get a web app running.

The release of Java support for Google AppEngine means more than just Java: it means lots of cool JVM languages! My personal favorite is Clojure, so when I and several colleagues got an opportunity to try out a pre-release version (thanks to a partnership between Google and ThoughtWorks) I immediately started trying it out. It turns out to be pretty easy and pretty great.

I'm going to walk you through using Clojure on Google AppEngine by stepping you through the creation of a Clojure/Compojure version of the Guestbook application from the Getting Started section of the AppEngine docs. As we go I'll introduce some bits of appengine-clj, a library I extracted along the way to keep anything that felt like boiler-plate code out of my app.

First you'll need to sign up (if you haven't already) and download the SDK.

Creating Your Application

To start, create a project directory and the basic project directory structure documented in the getting started section of Google's documentation. Here's an overview of what to create and what will go where.

project-root/
  src/
    [Clojure source code]
  war/
    [static files]
    WEB-INF/
      [config files]
      classes/
        [compiled classes from ../../src]
      lib/
        [jar dependencies]

Copy clojure.jar, clojure-contrib.jar, and compojure.jar into WEB-INF/lib. You'll also need to add appengine-api-XXX.jar from the SDK. I'm using fairly recent trunk versions of Clojure, clojure-contrib, and Compojure. (The easiest way to make sure you have compatible versions of these is to pull the latest Compojure source from github and then download their deps.zip and build Compojure against that.) If you want to use appengine-clj you can build it yourself from source or grab a prebuilt jar from the downloads section on Github.

Hello, World!

The entry point to your application will be a servlet class. Create a Clojure source file in your src directory and include a :gen-class directive to extend HttpServlet. Use Compojure's defroutes and defservice to create a HelloWorld.

src/guestbook/servlet.clj

(ns guestbook.servlet
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.http compojure.html))

(defroutes guestbook-app
  (GET "/"
    (html [:h1 "Hello, World!"])))

(defservice guestbook-app)

defroutes above creates a routing function that will respond to an HTML GET for the path "/", and the html function converts vectors to a string of HTML. If you're not familiar with Compojure, start here on the Compojure wiki.

Next create a web.xml with a servlet-mapping sending /* to your servlet class. Since Compojure handles URL routing, your application will have just this one mapping.

war/WEB-INF/web.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app 
    xmlns="http://java.sun.com/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">
  <display-name>Clojure Guestbook</display-name>
  <servlet>
    <servlet-name>gb</servlet-name>
    <servlet-class>guestbook.servlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>gb</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

Create an appengine-web.xml, putting your application ID into the application element.

war/WEB-INF/appengine-web.xml

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>[your application ID]</application>
  <version>1</version>
  <static-files />
  <resource-files />
</appengine-web-app>

Create an ant build.xml file that compiles your application to the classes directory.

build.xml

<project name="guestbook-clj" basedir="." default="compile">
  ... here you'll need to define project.classpath ...
  ... see GitHub for the full working example file ...
  <target name="compile" depends="...">
    <java classname="clojure.lang.Compile" classpathref="project.classpath" failonerror="true">
      <classpath path="${src.dir}" />
      <sysproperty key="clojure.compile.path" value="${classes.dir}" />
      <arg value="guestbook.servlet" />
    </java>
  </target>
</project>

I've left out some of the details, but you can find a full working version on Github.

At this point you should be able to run your "Hello, World" iteration of the application locally using the development appserver and deploy to appspot using the appcfg executable (or an Ant task if you prefer). Now it's just a matter of building your application using the tools Google AppEngine, Clojure, and Compojure make available to you.

The User Service

AppEngine has a simple API for dealing with user accounts. Let's greet logged in users by name. You'll need to import com.google.appengine.api.users.UserServiceFactory.

(ns ...
  (:import
    (com.google.appengine.api.users UserServiceFactory)))

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html [:h1 "Hello, " (if user (.getNickname user) "World") "!"]))))

But we have to let users log in to see this work. The UserService also exposes methods for creating login and logout URLs.

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL user-service "/") "sign in")]
        [:p (link-to (.createLogoutURL user-service "/") "sign out")]))))

Now you should be able to log into your application and be greeted by name. On the dev appserver, the login page will let you provide any username and check a box to indicate whether you should be logged in as an administrator for your application. On the appspot servers, you'll get a proper-looking Google Accounts login page. The argument to createLoginURL and createLogoutURL is the path or URL the user should be redirected to after logging in or out.

I've extracted the basic user-lookup calls into a Ring middleware function and put it into the appengine-clj.users namespace in appengine-clj. Here's what our servlet looks like using that.

(ns guestbook.servlet
  ... you no longer need to import UserServiceFactory ...
  (:require
    [appengine-clj.users :as users]))

(defroutes guestbook-app
  (GET "/"
    (let [user-info (request :appengine-clj/user-info)
          user (user-info :user)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL (user-info :user-service) "/") "sign in")]
        [:p (link-to (.createLogoutURL (user-info :user-service) "/") "sign out")]))))

(defservice (users/wrap-with-user-info guestbook-app))

It's about the same amount of code, it just looks a little more clojurey now. (I'm sorry: that's not a word.)

Datastore

Next let's collect guestbook entries and put them in the Datastore. AppEngine for Java has support for a couple of standard Java persistence APIs, JDO and JPA. But we'll use the lower-level datastore API, which seems a better fit for a dynamic language like Clojure (not to mention it doesn't require us to implement Java classes to persist).

The Java API for datastore is pretty simple, but conceptually it's different enough from a SQL database that it definitely takes some getting used to. (I for one am still figuring it out.)

Here's the sixty-second rundown of the bare essentials. The basic unit of persistence is the Entity, which has a Map of String-keyed properties. An Entity has a kind, which is a string denoting the type. (But keep in mind there's no schema here, so you can give any entity any properties.) An Entity is identified by a Key which is something more than a normal DBMS identifier because it can hold an Entity's association with a parent Entity. Besides using the Key, you can retrieve Entities with a Query, which searches either a single kind of entity or descendent entities of a single ancestor (or both, depending on which constructor you use) and can apply simple filtering and sorting.

The natural Clojurized form of an Entity seemed to be a map, so what I've started pulling out into appengine-clj.datastore is functions that allow Clojure code to work with an immutable map of keyword-keyed properties (plus :key and :kind) and have the library take care of translating into Entity objects. Currently there are just create and find methods, since that was all the basic guestbook needed. (But you know that Internet. I'll need a delete function before the week is out.)

Using appengine-clj.datastore, functions for creating and retrieving guestbook greetings are extremely simple.

(ns guestbook.greetings
  (:require [appengine-clj.datastore :as ds])
  (:import (com.google.appengine.api.datastore Query)))

(defn create [content author]
  (ds/create {:kind "Greeting" :author author :content content :date (java.util.Date.)}))

(defn find-all []
  (ds/find-all (doto (Query. "Greeting") (.addSort "date"))))

Note the creation of a com.google.appengine.api.datastore.Query (in find-all) that pulls back all Greetings and orders them by date. I've considered a couple of approaches for cleaning up creation of a query from Clojure, but I haven't decided between something that looks fairly idiomatic vs something that reads just like GQL. For the time being I'm sticking with the Java-interop style since even that is nicely terse and readable. Take a look at the tests for appengine-clj.datastore for more examples.

Speaking of which, this is a good time to mention that writing tests for datastore code is easy with appengine-clj.test-utils, which provides functions to set up an in-memory datastore. The dstest macro used there creates a fresh datastore for each test. If you're using a different testing framework or prefer different scoping, you can call ds-setup and ds-teardown yourself. (Do keep in mind that this is the development version of datastore, so we'll all need to keep an eye out for differences between that and the real Datastore service.)

HTML and Form Handling

Now that we've got our persistence straight (and tested), let's create a UI so users can sign the guestbook. At this point it's just plain Compojure code. We'll create one route to show the guestbook and a form to enter a greeting at "/" and another route for saving the greeting with a POST to "/sign".

Here's our function for signing the guestbook.

(defn sign-guestbook [params user]
  (greetings/create (params :content) (if user (.getNickname user)))
  (redirect-to "/"))

You can see there's very little to it. It takes the request parameters and a user, calls our greetings/create function, and redirects back to the guestbook.

The function for showing the guestbook is quite a bit more to swallow, since it includes our entire user interface.

(defn show-guestbook [{:keys [user user-service]}]
  (let [all-greetings (greetings/find-all)]
    (html [:html [:head [:title "Guestbook"]]
      [:body
        (if user
          [:p "Hello, " (.getNickname user) "! (You can "
            (link-to (.createLogoutURL user-service "/") "sign out")
            ".)"]
          [:p "Hello! (You can "
            (link-to (.createLoginURL user-service "/") "sign in")
            " to include your name with your greeting when you post.)"])
        (if (empty? all-greetings)
          [:p "The guestbook has no messages."]
          (map (fn [greeting]
            [:div
              [:p (if (greeting :author) [:strong (greeting :author)] "An anonymous guest") " wrote:"]
              [:blockquote (h (greeting :content))]])
            all-greetings))
        (form-to [POST "/sign"]
          [:div (text-area "content" "")]
          [:div (submit-button "Post Greeting")])]])))

It takes the user-info map, which it destructures to grab the user and UserService. It calls our greetings/find-all function to get the items to show and then uses Compojure's html helpers to create the document. For any real application you'd want to break the view down into smaller pieces to avoid such a huge nested chunk of vectors (or consider using another templating library like Enlive), but for this example I think it's easier to understand what's going on with the whole page in one function.

Finally here are the routes that wire it all together.

(defroutes guestbook-app
  (POST "/sign"
    (sign-guestbook params ((request :appengine-clj/user-info) :user)))
  (GET "/"
    (show-guestbook (request :appengine-clj/user-info))))

Here I'm using the :appengine-clj/user-info map that's been assoc'd to the request by the Ring middleware.

See the entire servlet file on GitHub, including some enhancements for styling and to see some other code to exercise Clojure features on AppEngine.

The Big Caveat

Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure.

First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures.

Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache.

More to Come

• I'll try and expand on this in the future with more write-ups, including a discussion of special handling for static files (which as of the version of the SDK I'm using works great on the appspot servers even with a /* servlet mapping but not on the local dev appserver, where servlet mappings win out over static files).
• If you'll sign my silly little guestbook on the appspot servers, I'd like to publish information on how many requests I got and how they performed.
• Google also provides Java APIs for caching, image manipulation, making HTTP requests, and email. I haven't even scratched the surface of those yet.
• With fresh support in AppEngine for scheduled tasks and upcoming support for task queues, there's more Clojure fun to be had.

Enjoy!

Update 7 September 2009: Late last week, the App Engine Team released version 1.2.5 of the SDK, including both XMPP (jabber instant messaging) support, which is brand new, and Task Queues, which had been available in the Python SDK but are now available for Java (and Clojure) applications.

clj-record Cleans Up Real Nice

Since clj-record started out as a sandbox for learning about Clojure macros, the project was a bit unstructured. Particularly weird was that the root source directory was the root of the project, so if you put that on your classpath, you also ended up with whatever random stuff was checked in at the top level on your classpath (for example TODO.txt and the script for running tests). That also gave me no good place to put dependencies, so you had to cobble together the right classpath on your own based on the instructions in the README.

I've now twisted the project into something more like a normal library layout. Source is rooted in /src. Dependencies are in /lib. Extra dependencies for running tests are in /test_lib. All this gets pulled together by Ant, which will run tests and spit out clj-record.jar (with ant all).

It was a painful moment starting work on that build.xml. Clojure would be a far better language for the build script—just about any language would be better than xml—but practicality won out: Clojure developers will have Ant installed. It was only a couple minutes before not having a real programming language there was an annoyance, and it will probably happen again. Hopefully once Programming Clojure is published, Stu Holloway will push forward with Lancet and we can all convert our build.xmls into build.cljs before long.

The Ant script also knows how to run a Nailgun server for VimClojure (with ant vim) that will have the source and tests on the classpath. (VimClojure is so badass.)

Certainly there's a lot to be said for the simplicity of just putting a thing on your classpath. You can still do that, just make it the project's /src directory instead of the root. At the moment the jar dependencies are checked into git, which could turn out to be a hassle for those who'd prefer to just put /src on their classpath. I could go to a separate download (as Compojure recently did) to keep the repository light. Let me know what you'd prefer.