RapidSMS at Away Day

Last weekend at the ThoughtWorks US Away Day we were joined by Merrick Schaefer from UNICEF Innovation and a couple developers who've worked extensively with RapidSMS. RapidSMS is an open source project providing a set of libraries and apps for interacting with phones. It's been used for a variety of purposes, mostly in Africa, where cell phone coverage is far better than electricity or land-line availability. One deployment they talked through in detail allows (relatively) untrained medical workers to use SMS messages to sumbit a few measurements to assess childhood malnutrition. In addition to recording the information to keep accurate and instantly available statistics, the system performs an immediate diagnosis and texts it back to the worker in the field.

Jeff Wishnie organized a RapidSMS code jam for those not in the mood to attend sessions, and Brian Guthrie and I spent most of Saturday pairing on a simple feature addition to one of the support applications. It was the first time either of us had touched Python for more than a few minutes, and the first time we'd touched Django at all, so it was an interesting learning experience. It was also extremely rewarding knowing that we were contributing to such a good cause.

If you sometimes find yourself in the mood for some coding but don't always have a personal project burning to get out, I'd highly encourage you to spend a little time familiarizing yourself with RapidSMS and maybe find something in their issue tracker you could tackle. I promise you'll feel great about it.

Interactive Clojure-AppEngine Development

If you like (1) interactive programming in Clojure from Emacs and (2) deploying to Google AppEngine, but you haven't yet managed to bring these two together, you should take a look at this post over at hackers-with-attitude, which walks you through setting up a pretty slick workflow.

SimURLator: Test Custom iPhone URL Schemes in the iPhone Simulator

Over the last few months ThoughtWorks has implemented several "Conference Connection" iPhone applications. They provide a view of the conference schedule, the ability to save a list of sessions you plan to attend, maps of the conferene site, a view of relevant Twitter posts, and some features specific to the various conferences.

The Twitter integration includes a feature to post, launching your prefered Twitter client (Tweetie, Echofon—formerly "TwitterFon"—or Twitterrific) using UIApplication's openURL: and the apps' custom URL schemes.

Relatively late in the testing cycle for another upcoming conference, we discovered an issue with selecting a Twitter client when you have more than one of the supported clients installed. Some of our testers were testing only in the iPhone Simulator, and because you can't install third-party apps in the Simulator, that feature had been undertested.

That was no good, so we created a sort of workaround: a stand-in app that takes the place of the third-party apps you can't install in the Simulator. SimURLator is now available on Github.

You configure SimURLator to respond to whatever custom URL schemes you want (by adding or removing URL schemes in SimURLator-Info.plist) and deploy it to your Simulator alongside your app. When SimURLator is launched via a URL by your app, it shows you the full URL. Obviously that's not as satisfying as seeing the third-party app launch and do the right thing, but it's a lot better than nothing.

You can also test your app's handling of its own custom URL schemes. Launch SimURLator and type in a URL that should launch your application. One button lets you verify that the OS reports the URL can be opened (with canOpenURL:); the other launches your app (with openURL:).

If there are other features you think would be useful, comment below. Better yet, fork it and show us.

Google Group for clj-record

The title pretty much says it all. There's now a Google Group for discussion of clj-record: clj-record-dev.

Renum 1.1: pretty instance-specific method definition

You could always define instance-specific behavior on Renum values the same way you can on any object in Ruby:

enum :FakeWord do
  Foo()
  Bar()
  Baz()
  
  def extra_stuff
    "no extra stuff here"
  end
end

def Foo.extra_stuff
  "here's the foo stuff"
end

That works fine, but it's a little ugly.

Starting this afternoon (with release 1.1) you can do this instead:

enum :FakeWord do
  Foo do
    def extra_stuff
      "here's the foo stuff"
    end
  end
  Bar()
  Baz()
  
  def extra_stuff
    "no extra stuff here"
  end
end

FakeWord::Foo will have its own implementation of extra_stuff while FakeWord::Bar and FakeWord::Baz will have the one defined inside the enum block. This was implemented by just instance_eval'ing the given block against the instance just after creating it. A def form in an instance_eval'd block ends up defining a singleton method, so it's equivalent to the uglier version above.

This ends up looking almost exactly like the Java enum equivalent for defining constant-specific class bodies.

Since the block is instance_eval'd, if you prefered you could do any initialization there instead of defining an init method. Depending on what you've got going on it may turn out to be more readable. Here's a contrived example from the spec translated to that style.

# init-with-args-style
enum :Size do
  Small("Really really tiny")
  Medium("Sort of in the middle")
  Large("Quite big")

  attr_reader :description

  def init description
    @description = description
  end
end

# instance-block-per-value-style
enum :Size do
  Small  { @description = "Really really tiny" }
  Medium { @description = "Sort of in the middle" }
  Large  { @description = "Quite big" }
  
  attr_reader :description
end

The latter is probably slightly easier to digest, but it's also an extremely simple case, so I'd expect your mileage to vary quite a bit. (Note that you can do both an init with arguments AND an associated block. Renum evals the block before calling init, so any instance-specific behavior is in place before your init runs.)

Anyway, comment, suggest, complain, fork, improve, and enjoy.

AR⊗TA

If you haven't already, you should check out Brian Marick's Agile reboot: Artisanal Retro-Futurism Crossed With Team Scale Anarcho-Syndicalism.

There's a video of his excellent presentation from Agile Roots if you don't feel like reading. There are also stickers you can ask for if you feel like spreading the word.

It's totally tangential to the point, but my favorite quote from the video is near the end of the questions: "...in my observation agile teams care more about delivering business value, often, than the business they're delivering it to."

A Quick Look at Azure, Microsoft's Cloud

Last week at a ThoughtWorks home-office gathering in New York we had a presentation by one of Microsoft's Architecture Evangelists about their in-the-works cloud offering, Azure. I have to say I was pretty impressed. (If not for the whole Windows thing, I'd probably be trying it out instead of writing about it right now. If you aren't as bothered by Windows as I am, you can register and try it yourself.) Here are some fairly random impressions, mostly based on comparison to the two other best-known cloud contenders.

Azure conceptually leans more towards Google AppEngine's "scale me up; scale me down; hide the details" approach than Amazon EC2's easy-to-manage-infrastructure-as-a-service model, though it currently sits somewhere between them. There's not (yet, at least) any automatic scaling up and down, which surprised me. However what you manually configure is just the number of instances of each application, which is far simpler than manually spinning up new instances of EC2 images and getting them into your cluster (or what-have-you). (Although if you've set up your infrastructure right, your EC2 scaling activities could be just as simple. EC2 just gives you the freedom to do it badly.)

There are two-and-a-half types of application: web apps, worker apps, and web+worker apps. The worker concept is very cool and not something you can do with AppEngine. (The closest thing is their support for scheduled tasks, but these are limited to scheduled URL hits no more often than once a minute with the same restrictions as a normal web request.) Apps all run on something pretty close to the normal .NET framework (minus normal file system access and presumably some other areas--I forgot to ask about threading.)

There are three kinds of storage.

• A file-system-like BLOB storage API, which involves saving individual blocks and then telling the system in what order to stitch them together. It seemed clunky, but I could see writing a simple abstraction over it for small stuff (where you'd usually only need one block) and for large stuff maybe being able to take advantage of already having some blocks uploaded when something gets interrupted and can resume later.
• A BigTable-like "table" storage API. It isn't as rich as AppEngine's BigTable-backed Datastore in that entity properties are restricted to very simple types (no lists, for example) and there's no built-in support for relationships of any kind (even the simple hierarchy of Datastore). (Note there's nothing to stop you from storing keys of other entities as property values and rolling your own awkward relationships.) Similar to Datastore on AppEngine, you have to think about how you want your data partitioned and make some under-informed guesses about slow queries across multiple partitions vs parallel queries each in their own partition. (On Azure, the key for each entity includes an app-generated token naming the partition in which it should reside, which is a bit simpler than Datastore's parentage-based entity groups.) The story on transactions (being limited to a single entity group) seems to be similar as well. There's also currently no indexing by anything other than key, though it sounded like they'll allow you to add at least one additional index per table before they go proper-live.
• Most interestingly, damn-near plain old SQL Server storage in the form of SQL Data Services, with just about every feature of MS SQL Server any normal application would use (except distributed transactions). I believe this is a separate service that you'll need to sign up for and eventually pay for separately. Regardless, it's a key element of the offering, because it provides a huge boost to the suggestion that you can migrate existing .NET applications into the cloud.

There are also persistent queues, which web apps can use to communicate to worker apps asynchronously. (I guess this qualifies as a fourth kind of storage.) They feature guaranteed delivery (with possible repeated delivery) of messages. This is very cool and not something AppEngine offers. (Even if you rolled your own queue-ish thing in datastore, there's no facility to consume messages intelligently.)

There are REST APIs (and SOAP as well, I think) for just about everything, so you have a lot of options for architectures (using heterogeneous technologies) that run partly on-premisis and partly in the Azure cloud.

There are a bunch of other services you can hook into, including ".NET Services" (low-level technical stuff including a Service Bus, Access Control, and a Workflow Service) and "Live Services" (higher-level end-user-meaningful services such as Contacts, Windows Live ID, and assorted Bing stuff).

There's also, surprisingly, the ability to run unmanaged code in your .NET application, AND some form of support for non-.NET stuff via IIS's FastCGI support.

Anyway, it looks like it'll be a compelling offering, competitive with EC2-hosted Windows, and thanks to SQL Data Services, much closer to allowing you to put a normal .NET web app into the cloud than AppEngine can do for Java web apps. For any organization willing to trust Microsoft with its infrastructure, Azure is a good reason to be excited. For the rest of the world, carefully executed EC2 still looks like the best bet for the Enterprise in the cloud.

Renum Moved to Github

Chris O'Meara recently contributed a fix to make constantize_attribute handle nil values sanely. (Did you know "".constantize # => Object? I was sure surprised.)

He's using constantize_attribute to persist renum enumeration values in ActiveRecord models, which is the reason I wrote it in the first place. The reminder that these things do get used here and there inspired me to give renum a little late Spring cleaning.

Renum now lives on github and is set up to be gem-packaged with jeweler, which is a lot more elegant than the previous solution (at least in terms of how much extra stuff it puts into source control). Release 1.0.2 is tagged, but it's functionally the same as 1.0.1 (it just lives in a better SCM and has less packaging cruft).

clj-record now has find-by-sql

I don't know why I didn't do this 3 months ago (or 5), but this morning I extracted find-by-sql from find-records so you can do arbitrary queries and still have them run through the callbacks defined for your model.

Specifying Column Order in ActiveRecord Migrations

ActiveRecord migrations don't allow you to control column order when adding or changing columns. You can always just

execute "ALTER TABLE ..." # but it's ugly and hard to remember the details.

If you're willing to give up vendor neutrality, you can have the feature in pretty-Ruby-migration syntax with a small effort. Here's an implementation for MySQL that adds options :first and :after to add_column and change_column.

ActiveRecord::ConnectionAdapters::MysqlAdapter.class_eval do
  def add_column_options!(sql, options)
    super
    if options[:after]
      sql << " AFTER #{quote_column_name(options[:after])}"
    elsif options[:first]
      sql << " FIRST"
    end
  end
end

(It's also available in this gist if you'd like to suggest a cleaner way.)

With that in place,

a.add_column :foos, :bar, :string, :after => :baz

will execute

ALTER TABLE `foos` ADD `bar` varchar(255) AFTER `baz`

and

a.add_column :foos, :bar, :string, :first => true

will execute

ALTER TABLE `foos` ADD `bar` varchar(255) FIRST

Love that Ruby!

Now this is a monkey-patch, but as monkey-patches go it's pretty innocuous. Rather than redefining a method that's already implemented in MysqlAdapter (and as with many monkey-patches copy-pasting the existing implementation so we can modify it), we're defining a new override of a method (albeit undocumented) inherited from the SchemaStatements module (via AbstractAdapter). If you wanted to, you could subclass MysqlAdapter and then configure Rails to use your own adapter, but I suspect that would be more expensive to own than just adding this method to MysqlAdapter. (Beware: there are many useful Rails-provided rake tasks that look at your adapter string to decide whether to be useful or to spit in your face.)

Clojure on Google AppEngine

Note: This post is quite old, and Compojure has changed (and gotten quite a bit bigger) since it was written. The basic steps for setting up a Clojure GAE app haven't changed much, but the Compojure bits may no longer be so simple. You can always just AOT-compile a Clojure HttpServlet to get a web app running.

The release of Java support for Google AppEngine means more than just Java: it means lots of cool JVM languages! My personal favorite is Clojure, so when I and several colleagues got an opportunity to try out a pre-release version (thanks to a partnership between Google and ThoughtWorks) I immediately started trying it out. It turns out to be pretty easy and pretty great.

I'm going to walk you through using Clojure on Google AppEngine by stepping you through the creation of a Clojure/Compojure version of the Guestbook application from the Getting Started section of the AppEngine docs. As we go I'll introduce some bits of appengine-clj, a library I extracted along the way to keep anything that felt like boiler-plate code out of my app.

First you'll need to sign up (if you haven't already) and download the SDK.

Creating Your Application

To start, create a project directory and the basic project directory structure documented in the getting started section of Google's documentation. Here's an overview of what to create and what will go where.

project-root/
  src/
    [Clojure source code]
  war/
    [static files]
    WEB-INF/
      [config files]
      classes/
        [compiled classes from ../../src]
      lib/
        [jar dependencies]

Copy clojure.jar, clojure-contrib.jar, and compojure.jar into WEB-INF/lib. You'll also need to add appengine-api-XXX.jar from the SDK. I'm using fairly recent trunk versions of Clojure, clojure-contrib, and Compojure. (The easiest way to make sure you have compatible versions of these is to pull the latest Compojure source from github and then download their deps.zip and build Compojure against that.) If you want to use appengine-clj you can build it yourself from source or grab a prebuilt jar from the downloads section on Github.

Hello, World!

The entry point to your application will be a servlet class. Create a Clojure source file in your src directory and include a :gen-class directive to extend HttpServlet. Use Compojure's defroutes and defservice to create a HelloWorld.

src/guestbook/servlet.clj

(ns guestbook.servlet
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.http compojure.html))

(defroutes guestbook-app
  (GET "/"
    (html [:h1 "Hello, World!"])))

(defservice guestbook-app)

defroutes above creates a routing function that will respond to an HTML GET for the path "/", and the html function converts vectors to a string of HTML. If you're not familiar with Compojure, start here on the Compojure wiki.

Next create a web.xml with a servlet-mapping sending /* to your servlet class. Since Compojure handles URL routing, your application will have just this one mapping.

war/WEB-INF/web.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app 
    xmlns="http://java.sun.com/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">
  <display-name>Clojure Guestbook</display-name>
  <servlet>
    <servlet-name>gb</servlet-name>
    <servlet-class>guestbook.servlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>gb</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

Create an appengine-web.xml, putting your application ID into the application element.

war/WEB-INF/appengine-web.xml

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>[your application ID]</application>
  <version>1</version>
  <static-files />
  <resource-files />
</appengine-web-app>

Create an ant build.xml file that compiles your application to the classes directory.

build.xml

<project name="guestbook-clj" basedir="." default="compile">
  ... here you'll need to define project.classpath ...
  ... see GitHub for the full working example file ...
  <target name="compile" depends="...">
    <java classname="clojure.lang.Compile" classpathref="project.classpath" failonerror="true">
      <classpath path="${src.dir}" />
      <sysproperty key="clojure.compile.path" value="${classes.dir}" />
      <arg value="guestbook.servlet" />
    </java>
  </target>
</project>

I've left out some of the details, but you can find a full working version on Github.

At this point you should be able to run your "Hello, World" iteration of the application locally using the development appserver and deploy to appspot using the appcfg executable (or an Ant task if you prefer). Now it's just a matter of building your application using the tools Google AppEngine, Clojure, and Compojure make available to you.

The User Service

AppEngine has a simple API for dealing with user accounts. Let's greet logged in users by name. You'll need to import com.google.appengine.api.users.UserServiceFactory.

(ns ...
  (:import
    (com.google.appengine.api.users UserServiceFactory)))

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html [:h1 "Hello, " (if user (.getNickname user) "World") "!"]))))

But we have to let users log in to see this work. The UserService also exposes methods for creating login and logout URLs.

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL user-service "/") "sign in")]
        [:p (link-to (.createLogoutURL user-service "/") "sign out")]))))

Now you should be able to log into your application and be greeted by name. On the dev appserver, the login page will let you provide any username and check a box to indicate whether you should be logged in as an administrator for your application. On the appspot servers, you'll get a proper-looking Google Accounts login page. The argument to createLoginURL and createLogoutURL is the path or URL the user should be redirected to after logging in or out.

I've extracted the basic user-lookup calls into a Ring middleware function and put it into the appengine-clj.users namespace in appengine-clj. Here's what our servlet looks like using that.

(ns guestbook.servlet
  ... you no longer need to import UserServiceFactory ...
  (:require
    [appengine-clj.users :as users]))

(defroutes guestbook-app
  (GET "/"
    (let [user-info (request :appengine-clj/user-info)
          user (user-info :user)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL (user-info :user-service) "/") "sign in")]
        [:p (link-to (.createLogoutURL (user-info :user-service) "/") "sign out")]))))

(defservice (users/wrap-with-user-info guestbook-app))

It's about the same amount of code, it just looks a little more clojurey now. (I'm sorry: that's not a word.)

Datastore

Next let's collect guestbook entries and put them in the Datastore. AppEngine for Java has support for a couple of standard Java persistence APIs, JDO and JPA. But we'll use the lower-level datastore API, which seems a better fit for a dynamic language like Clojure (not to mention it doesn't require us to implement Java classes to persist).

The Java API for datastore is pretty simple, but conceptually it's different enough from a SQL database that it definitely takes some getting used to. (I for one am still figuring it out.)

Here's the sixty-second rundown of the bare essentials. The basic unit of persistence is the Entity, which has a Map of String-keyed properties. An Entity has a kind, which is a string denoting the type. (But keep in mind there's no schema here, so you can give any entity any properties.) An Entity is identified by a Key which is something more than a normal DBMS identifier because it can hold an Entity's association with a parent Entity. Besides using the Key, you can retrieve Entities with a Query, which searches either a single kind of entity or descendent entities of a single ancestor (or both, depending on which constructor you use) and can apply simple filtering and sorting.

The natural Clojurized form of an Entity seemed to be a map, so what I've started pulling out into appengine-clj.datastore is functions that allow Clojure code to work with an immutable map of keyword-keyed properties (plus :key and :kind) and have the library take care of translating into Entity objects. Currently there are just create and find methods, since that was all the basic guestbook needed. (But you know that Internet. I'll need a delete function before the week is out.)

Using appengine-clj.datastore, functions for creating and retrieving guestbook greetings are extremely simple.

(ns guestbook.greetings
  (:require [appengine-clj.datastore :as ds])
  (:import (com.google.appengine.api.datastore Query)))

(defn create [content author]
  (ds/create {:kind "Greeting" :author author :content content :date (java.util.Date.)}))

(defn find-all []
  (ds/find-all (doto (Query. "Greeting") (.addSort "date"))))

Note the creation of a com.google.appengine.api.datastore.Query (in find-all) that pulls back all Greetings and orders them by date. I've considered a couple of approaches for cleaning up creation of a query from Clojure, but I haven't decided between something that looks fairly idiomatic vs something that reads just like GQL. For the time being I'm sticking with the Java-interop style since even that is nicely terse and readable. Take a look at the tests for appengine-clj.datastore for more examples.

Speaking of which, this is a good time to mention that writing tests for datastore code is easy with appengine-clj.test-utils, which provides functions to set up an in-memory datastore. The dstest macro used there creates a fresh datastore for each test. If you're using a different testing framework or prefer different scoping, you can call ds-setup and ds-teardown yourself. (Do keep in mind that this is the development version of datastore, so we'll all need to keep an eye out for differences between that and the real Datastore service.)

HTML and Form Handling

Now that we've got our persistence straight (and tested), let's create a UI so users can sign the guestbook. At this point it's just plain Compojure code. We'll create one route to show the guestbook and a form to enter a greeting at "/" and another route for saving the greeting with a POST to "/sign".

Here's our function for signing the guestbook.

(defn sign-guestbook [params user]
  (greetings/create (params :content) (if user (.getNickname user)))
  (redirect-to "/"))

You can see there's very little to it. It takes the request parameters and a user, calls our greetings/create function, and redirects back to the guestbook.

The function for showing the guestbook is quite a bit more to swallow, since it includes our entire user interface.

(defn show-guestbook [{:keys [user user-service]}]
  (let [all-greetings (greetings/find-all)]
    (html [:html [:head [:title "Guestbook"]]
      [:body
        (if user
          [:p "Hello, " (.getNickname user) "! (You can "
            (link-to (.createLogoutURL user-service "/") "sign out")
            ".)"]
          [:p "Hello! (You can "
            (link-to (.createLoginURL user-service "/") "sign in")
            " to include your name with your greeting when you post.)"])
        (if (empty? all-greetings)
          [:p "The guestbook has no messages."]
          (map (fn [greeting]
            [:div
              [:p (if (greeting :author) [:strong (greeting :author)] "An anonymous guest") " wrote:"]
              [:blockquote (h (greeting :content))]])
            all-greetings))
        (form-to [POST "/sign"]
          [:div (text-area "content" "")]
          [:div (submit-button "Post Greeting")])]])))

It takes the user-info map, which it destructures to grab the user and UserService. It calls our greetings/find-all function to get the items to show and then uses Compojure's html helpers to create the document. For any real application you'd want to break the view down into smaller pieces to avoid such a huge nested chunk of vectors (or consider using another templating library like Enlive), but for this example I think it's easier to understand what's going on with the whole page in one function.

Finally here are the routes that wire it all together.

(defroutes guestbook-app
  (POST "/sign"
    (sign-guestbook params ((request :appengine-clj/user-info) :user)))
  (GET "/"
    (show-guestbook (request :appengine-clj/user-info))))

Here I'm using the :appengine-clj/user-info map that's been assoc'd to the request by the Ring middleware.

See the entire servlet file on GitHub, including some enhancements for styling and to see some other code to exercise Clojure features on AppEngine.

The Big Caveat

Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure.

First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures.

Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache.

More to Come

• I'll try and expand on this in the future with more write-ups, including a discussion of special handling for static files (which as of the version of the SDK I'm using works great on the appspot servers even with a /* servlet mapping but not on the local dev appserver, where servlet mappings win out over static files).
• If you'll sign my silly little guestbook on the appspot servers, I'd like to publish information on how many requests I got and how they performed.
• Google also provides Java APIs for caching, image manipulation, making HTTP requests, and email. I haven't even scratched the surface of those yet.
• With fresh support in AppEngine for scheduled tasks and upcoming support for task queues, there's more Clojure fun to be had.

Enjoy!

Update 7 September 2009: Late last week, the App Engine Team released version 1.2.5 of the SDK, including both XMPP (jabber instant messaging) support, which is brand new, and Task Queues, which had been available in the Python SDK but are now available for Java (and Clojure) applications.

clj-record Cleans Up Real Nice

Since clj-record started out as a sandbox for learning about Clojure macros, the project was a bit unstructured. Particularly weird was that the root source directory was the root of the project, so if you put that on your classpath, you also ended up with whatever random stuff was checked in at the top level on your classpath (for example TODO.txt and the script for running tests). That also gave me no good place to put dependencies, so you had to cobble together the right classpath on your own based on the instructions in the README.

I've now twisted the project into something more like a normal library layout. Source is rooted in /src. Dependencies are in /lib. Extra dependencies for running tests are in /test_lib. All this gets pulled together by Ant, which will run tests and spit out clj-record.jar (with ant all).

It was a painful moment starting work on that build.xml. Clojure would be a far better language for the build script—just about any language would be better than xml—but practicality won out: Clojure developers will have Ant installed. It was only a couple minutes before not having a real programming language there was an annoyance, and it will probably happen again. Hopefully once Programming Clojure is published, Stu Holloway will push forward with Lancet and we can all convert our build.xmls into build.cljs before long.

The Ant script also knows how to run a Nailgun server for VimClojure (with ant vim) that will have the source and tests on the classpath. (VimClojure is so badass.)

Certainly there's a lot to be said for the simplicity of just putting a thing on your classpath. You can still do that, just make it the project's /src directory instead of the root. At the moment the jar dependencies are checked into git, which could turn out to be a hassle for those who'd prefer to just put /src on their classpath. I could go to a separate download (as Compojure recently did) to keep the repository light. Let me know what you'd prefer.

Attribute Serialization for clj-record

I've just pushed clj-record support for attribute serialization to github. This uses clj-record callbacks and clojure.core's pr and read functions to allow you to store many standard Clojure and Java types as record attributes in varchar columns of your database.

I'm not crazy about calling this feature "serialization" since it's not using Java's serialization mechanism (though I'm very happy about Clojure's fairly human-readable pr/prn/read format). If you have thoughts on a better name, let me hear them.

Fancy Queries for clj-record

Thanks to another contribution from Tunde Ashafa, clj-record now has support for more advanced queries using functions in the new clj-record.query namespace.

Previously you could do this:

(manufacturer/find-records {:grade 98})
; or
(manufacturer/find-records {:name "Humedai Motors"})

Now you can do this:

; having required [clj-record.query :as q]
(manufacturer/find-records {:grade (q/greater-than 90)})
; or
(manufacturer/find-records {:name (q/like "Hume%")})

See the tests for more examples of what it does. See the source for how it's implemented. And please: recommend improvements.

clj-record support for MySQL and PostgreSQL

Thanks to pull requests from a couple of helpful github users (vishnu and tashafa), clj-record should now work with both MySQL and PostgreSQL.

The Ins and Outs of clj-record Validations

One of the first features I tackled in clj-record was a declarative validation API. Here I'll look at how it works, both in terms of how to use it and how it's implemented.

How to declare validations

The init-model form can include an option-group that opens with the keyword :validation like this.

(clj-record.core/init-model
  (:validation
    (:nickname "Nickname is required." #(not (empty? %)))))

Following :validation you provide any number of list forms with the attribute name as a keyword, the message that should be reported if the validation fails, and the function that implements the validation. It must take one argument, the value being validated, and return logical true or false. The function can be defined inline, as shown above, or a reference to a function defined elsewhere. (That is, even though init-model is a macro, your code will be evaluated as you expect.)

Built-in validations are provided in clj-record.validation.built-ins.

; assuming you've required [clj-record.validation.built-ins :as valid]
(clj-record.core/init-model
  (:validation
    (:nickname
      "Nickname must be three to fifteen letters."
      (valid/match #"^[A-Za-z]{3,15}$"))))

clj-record.validation.built-ins/match is a higher order function that takes a regular expression pattern and returns a validation function that passes only if the value matches the pattern (in whole or in part, so use ^ and $ in your pattern if you want to match the whole string).

Some of the functions in clj-record.validation.built-ins are, like match, higher order functions that return validation functions. Others, for example numeric?, are simple validation functions that you refer to directly.

(clj-record.core/init-model
  (:validation
    (:age "Age ain't nothing but a number." valid/numeric?)))

Since they're just normal functions, the built-ins can be combined just like any other functions.

(clj-record.core/init-model
  (:validation
    (:age "Age ain't nothing but a number (or nil)."
      #(or (nil? %) (valid/numeric? %)))))

How to run validations

ActiveRecord's validation API is nice looking in the normal case but conceptually a bit nasty. The valid? method looks like a pure predicate, but it mutates the model behind the scenes, clearing and re-populating its errors collection each time it's called. Since immutability is highly valued in functional languages, it was clear a similar approach wouldn't be appropriate for clj-record.

Model validations in clj-record are run using the validate function of the model namespace, which returns a validation-result. (If you can suggest a better name for that, please do.) The result can be inspected with the predicate clj-record.validation/valid? and messages for an attribute can be retrieved with clj-record.validation/messages-for, which takes the attribute name keyword and returns a collection of messages.

How validations work

The details of how validations are implemented shouldn't be relevant for normal use, but since clj-record is very young, it's pretty likely you'll need to look at the internals. If you're new to Clojure (or LISP macros in general) perhaps you'll also find this a useful case-study. I'm also interested in suggestions for improvements (to both the internals and the public-facing API).

First let's look at how option-groups in the init-model form work. The init-model macro uses the opening keyword of each option-group to look up a namespace. The validation option-group opens with :validation, so the namespace will be clj-record.validation.

It then calls a function in that namespace called expand-init-option once for each option-form in the option-group. (In the first example above, the only option-form is "(:nickname "Nickname is required." #(not (empty? %)))," but there can be more.) expand-init-option takes as arguments the model name (as a string) followed by whatever appeared in the option-form and returns a form that will appear in the expansion of init-model.

As you can guess from the examples above, clj-record.validation/expand-init-option takes as arguments the model-name followed by the (unevaluated) attribute name, message, and validation function. It returns a syntax-quoted form that calls clj-record.validation/add-validation with those same arguments verbatim, which in turn adds a validation to the (mutable) model metadata of your model.

So this:

; in a model namespace called com.example.model.foo
(clj-record.core/init-model
  (:validation
    (:nickname "Nickname is required." #(not (empty? %)))
    (:age "Age ain't nothing but a number." valid/numeric?)))

is equivalent to this:

; in a model namespace called com.example.model.foo
(clj-record.core/init-model)
(clj-record.validation/add-validation "foo"
  :nickname "Nickname is required." #(not (empty? %)))
(clj-record.validation/add-validation "foo"
  :age "Age ain't nothing but a number." valid/numeric?)

since the init-model macro will expand to a form including those exact add-validation calls.

Notice that very little work is happening in the macro-expansion. Keeping the macro layer very thin yields a number of benefits.

• First, it makes it trivial to allow references in the option-forms to work as you'd expect. In an earlier version of the implementation, model-metadata was built up at macro-expansion time, but it turned out I had to jump through extra eval hoops to get symbols to resolve to the right thing.
• Second, it keeps things working with AOT (ahead-of-time compilation). After jumping through the eval hoops, I decided to test a pre-compiled model class and realized that the in-memory model-metadata built up at macro-expansion time didn't exist, because macros had been expanded back when I compiled. Oops!
• Third, it makes testing much easier. I can define a funny validation for just one test and don't have to worry about keeping it passing in other tests. At the end of a test I just repoint a mutable ref back at the old value, and the model-metadata is back to its original shape. (I actually haven't yet used that technique for validation tests but will probably start soon. I do already use it in the tests for callbacks.)
• Finally, it reduces conceptual overhead. Macros are complicated. Anything you can do to make sure macro-expansion is just a simple conversion into something unmagical helps reduce head scratching.

Internally the validations are stored in model-metadata as a vector, since I wanted to run them in the order they're defined (for no real reason other than predictability). The validation result returned by validate is just a clojure hash map, but I provided valid? and messages-for so the implementation is abstracted. (Currently valid? is just empty? and messages-for is just get, but perhaps they'll get fancier later.)

So much for validations. Next week, maybe callbacks ... or maybe I'll actually do some coding.

RSpec's New Implicit Subjects

In case you missed the announcement, RSpec now supports this badassery.

describe Person do
  describe "born 19 years ago" do
    subject { Person.new(:birthdate => 19.years.ago }
    it { should be_eligible_to_vote }
    it { should be_eligible_to_enlist }
    it { should_not be_eligible_to_drink }
  end
end

clj-record: ActiveRecord for Clojure

I've spent a chunk of the last month (and an absurd amount of the last four days) working on clj-record: an ActiveRecord-like library written in the Clojure programming language. (Clojure is a LISP dialect with badass concurrency features for the JVM.) clj-record aims to provide an idiomatic "functional" API for persistence with many of the features we've come to appreciate from ActiveRecord, currently including validation, (somewhat) convenient associations, and (some) callbacks.

(Persistence requires side-effects, so it can't really be functional, but it can be idiomatic and leverage a great deal of power from being written in a functional language.)

Here's what a model definition looks like:

(ns clj-record.test.model.manufacturer
  (:require clj-record.boot))

(clj-record.core/init-model
  (:associations
    (has-many products))
  (:validation
    (:name "empty!" #(not (empty? %)))
    (:founded "must be numeric" #(or (nil? %) (not (re-find #"\D" %))))))

Coming soon:

• Even more convenient associations
• Even more callbacks
• Serialization of data structures as attribute values
• A fancy query API
• Other cool stuff

Check it out!