Why Not To Use My Library clj-record

In 2008 I discovered Clojure, and I was extremely excited to finally have an easy entry-way to Lisp. Pretty early on I wanted to get my head around one of Lisp's killer features: macros. A little playing around in the REPL was easy but not satisfying: I wanted to build something powerful.

Having spent a few years doing Ruby on Rails, I thought ActiveRecord's macro-like capabilities (has_many and whatnot) would be a good goal. So I built clj-record.

In retrospect, I went too far trying to build in Rails-like magic, so I want to point out the design flaws that I recommend you avoid in your own work and recommend that you choose something other than clj-record for your Clojure projects that use an RDBMS.

The API of clj-record is all in one macro: clj-record.core/init-model. It commits a few macro sins.

• It looks at the namespace from which you call it to find the name of your model.
• It intentionally captures a db var from your namespace.
• It defs a bunch of fns in your namespace (most of which are just partially applied versions of clj-record.core fns).

So you do this:

and that turns into something like this (leaving out the ns):

This provided really good practice (and an example I've gone back to more than once) for quoting, syntax quoting, unquoting, and unquote-splicing.

But this sucks!

What init-model should do is return a data-structure that you then pass to clj-record's find, insert, and update fns. In addition to being simpler to understand than the hidden registry of model metadata that clj-record has to maintain, it would make it more natural to write generic, composable code, since the model itself would be a value.

If that were all init-model needed to do, the code (both inside clj-record and user code) would probably be even simpler if there were no macro at all. A set of functions could enhance the model data-structure as needed so that even model setup code could be composed like any other series of fn-calls.

Maybe it would be worthwhile having a macro that went something like this:

This macro would do a def, but it's completely intuitive what that def is. The third "sin" on my list above was about defining a bunch of fns in your namespace. I call this a sin because it's not clear from reading the code that there are a bunch of vars defined, making both discovery and debugging more difficult. There may be cases where "hidden defs" are justified by performance gains or code reduction, but the fns defined by init-model don't pass that test (because (clj-record.core/find-records widget attributes) is no worse and more easily composed with other generic model-processing fns than (my.model.widget/find-records attributes)).

If no one else were writing libraries, I'd try to remedy this in the next major revision of clj-record, but someone else has done this work (or something very much like it) already. Chris Granger wrote korma, and it offers similar functionality in a saner way. I'm also not using Clojure with a SQL database, so I wouldn't be able to eat my own dogfood.

If you're modeling entities in a relational database, I'd recommend korma over clj-record. There may be even better options for you (including just using clojure.java.jdbc directly), so look around.

If you're writing macros, don't capture vars, don't make their functionality depend on globals like *ns*, and consider alternatives to creating vars in the caller's namespace. While you're at it, consider whether the job would be better done with plain old functions.

FluentTime: a fluent interface in C# for working with dates and times

After working fairly late last night I was too wound up to go to sleep when I got home. Something possessed me to start creating a little C# library providing a fluent interface for working with DateTimes, DateTimeOffsets, and TimeSpans inspired by ActiveSupport's active_support/core_ext/numeric/time and related extensions.

After about 3.Hours(30.Minutes()) the straightforward part had come together. You can do stuff like this with it:

var toothBrushingTime = 10.Minutes().Before(bedTime);
var lightsOutForReal = 1.Hour(30.Minutes()).After(bedTime);
...
Assert.That(actual, Is.LessThan(50.Milliseconds().After(expected)));

These built in .NET types are pretty easy to work with and made the implementation very easy (except for the bit where I discovered that TimeSpan.FromMilliseconds takes a double but only has whole-millisecond precision).

This morning I put it up on Github. You can find FluentTime here.

The one more complex thing I'd still like to add is a variable-length variation on TimeSpan for handling units like months and years. I'm interested to see how clean I can keep things when my type will only take on a specific value based on the date-time value it's being added to or subtracted from. That ought to be a good opportunity for exploring operator overloading. (Unsurprisingly that hasn't come up in my day job just yet.)

SimURLator: Test Custom iPhone URL Schemes in the iPhone Simulator

Over the last few months ThoughtWorks has implemented several "Conference Connection" iPhone applications. They provide a view of the conference schedule, the ability to save a list of sessions you plan to attend, maps of the conferene site, a view of relevant Twitter posts, and some features specific to the various conferences.

The Twitter integration includes a feature to post, launching your prefered Twitter client (Tweetie, Echofon—formerly "TwitterFon"—or Twitterrific) using UIApplication's openURL: and the apps' custom URL schemes.

Relatively late in the testing cycle for another upcoming conference, we discovered an issue with selecting a Twitter client when you have more than one of the supported clients installed. Some of our testers were testing only in the iPhone Simulator, and because you can't install third-party apps in the Simulator, that feature had been undertested.

That was no good, so we created a sort of workaround: a stand-in app that takes the place of the third-party apps you can't install in the Simulator. SimURLator is now available on Github.

You configure SimURLator to respond to whatever custom URL schemes you want (by adding or removing URL schemes in SimURLator-Info.plist) and deploy it to your Simulator alongside your app. When SimURLator is launched via a URL by your app, it shows you the full URL. Obviously that's not as satisfying as seeing the third-party app launch and do the right thing, but it's a lot better than nothing.

You can also test your app's handling of its own custom URL schemes. Launch SimURLator and type in a URL that should launch your application. One button lets you verify that the OS reports the URL can be opened (with canOpenURL:); the other launches your app (with openURL:).

If there are other features you think would be useful, comment below. Better yet, fork it and show us.

Google Group for clj-record

The title pretty much says it all. There's now a Google Group for discussion of clj-record: clj-record-dev.

Renum 1.1: pretty instance-specific method definition

You could always define instance-specific behavior on Renum values the same way you can on any object in Ruby:

enum :FakeWord do
  Foo()
  Bar()
  Baz()
  
  def extra_stuff
    "no extra stuff here"
  end
end

def Foo.extra_stuff
  "here's the foo stuff"
end

That works fine, but it's a little ugly.

Starting this afternoon (with release 1.1) you can do this instead:

enum :FakeWord do
  Foo do
    def extra_stuff
      "here's the foo stuff"
    end
  end
  Bar()
  Baz()
  
  def extra_stuff
    "no extra stuff here"
  end
end

FakeWord::Foo will have its own implementation of extra_stuff while FakeWord::Bar and FakeWord::Baz will have the one defined inside the enum block. This was implemented by just instance_eval'ing the given block against the instance just after creating it. A def form in an instance_eval'd block ends up defining a singleton method, so it's equivalent to the uglier version above.

This ends up looking almost exactly like the Java enum equivalent for defining constant-specific class bodies.

Since the block is instance_eval'd, if you prefered you could do any initialization there instead of defining an init method. Depending on what you've got going on it may turn out to be more readable. Here's a contrived example from the spec translated to that style.

# init-with-args-style
enum :Size do
  Small("Really really tiny")
  Medium("Sort of in the middle")
  Large("Quite big")

  attr_reader :description

  def init description
    @description = description
  end
end

# instance-block-per-value-style
enum :Size do
  Small  { @description = "Really really tiny" }
  Medium { @description = "Sort of in the middle" }
  Large  { @description = "Quite big" }
  
  attr_reader :description
end

The latter is probably slightly easier to digest, but it's also an extremely simple case, so I'd expect your mileage to vary quite a bit. (Note that you can do both an init with arguments AND an associated block. Renum evals the block before calling init, so any instance-specific behavior is in place before your init runs.)

Anyway, comment, suggest, complain, fork, improve, and enjoy.

Renum Moved to Github

Chris O'Meara recently contributed a fix to make constantize_attribute handle nil values sanely. (Did you know "".constantize # => Object? I was sure surprised.)

He's using constantize_attribute to persist renum enumeration values in ActiveRecord models, which is the reason I wrote it in the first place. The reminder that these things do get used here and there inspired me to give renum a little late Spring cleaning.

Renum now lives on github and is set up to be gem-packaged with jeweler, which is a lot more elegant than the previous solution (at least in terms of how much extra stuff it puts into source control). Release 1.0.2 is tagged, but it's functionally the same as 1.0.1 (it just lives in a better SCM and has less packaging cruft).

clj-record now has find-by-sql

I don't know why I didn't do this 3 months ago (or 5), but this morning I extracted find-by-sql from find-records so you can do arbitrary queries and still have them run through the callbacks defined for your model.

Clojure on Google AppEngine

Note: This post is quite old, and Compojure has changed (and gotten quite a bit bigger) since it was written. The basic steps for setting up a Clojure GAE app haven't changed much, but the Compojure bits may no longer be so simple. You can always just AOT-compile a Clojure HttpServlet to get a web app running.

The release of Java support for Google AppEngine means more than just Java: it means lots of cool JVM languages! My personal favorite is Clojure, so when I and several colleagues got an opportunity to try out a pre-release version (thanks to a partnership between Google and ThoughtWorks) I immediately started trying it out. It turns out to be pretty easy and pretty great.

I'm going to walk you through using Clojure on Google AppEngine by stepping you through the creation of a Clojure/Compojure version of the Guestbook application from the Getting Started section of the AppEngine docs. As we go I'll introduce some bits of appengine-clj, a library I extracted along the way to keep anything that felt like boiler-plate code out of my app.

First you'll need to sign up (if you haven't already) and download the SDK.

Creating Your Application

To start, create a project directory and the basic project directory structure documented in the getting started section of Google's documentation. Here's an overview of what to create and what will go where.

project-root/
  src/
    [Clojure source code]
  war/
    [static files]
    WEB-INF/
      [config files]
      classes/
        [compiled classes from ../../src]
      lib/
        [jar dependencies]

Copy clojure.jar, clojure-contrib.jar, and compojure.jar into WEB-INF/lib. You'll also need to add appengine-api-XXX.jar from the SDK. I'm using fairly recent trunk versions of Clojure, clojure-contrib, and Compojure. (The easiest way to make sure you have compatible versions of these is to pull the latest Compojure source from github and then download their deps.zip and build Compojure against that.) If you want to use appengine-clj you can build it yourself from source or grab a prebuilt jar from the downloads section on Github.

Hello, World!

The entry point to your application will be a servlet class. Create a Clojure source file in your src directory and include a :gen-class directive to extend HttpServlet. Use Compojure's defroutes and defservice to create a HelloWorld.

src/guestbook/servlet.clj

(ns guestbook.servlet
  (:gen-class :extends javax.servlet.http.HttpServlet)
  (:use compojure.http compojure.html))

(defroutes guestbook-app
  (GET "/"
    (html [:h1 "Hello, World!"])))

(defservice guestbook-app)

defroutes above creates a routing function that will respond to an HTML GET for the path "/", and the html function converts vectors to a string of HTML. If you're not familiar with Compojure, start here on the Compojure wiki.

Next create a web.xml with a servlet-mapping sending /* to your servlet class. Since Compojure handles URL routing, your application will have just this one mapping.

war/WEB-INF/web.xml

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app 
    xmlns="http://java.sun.com/xml/ns/javaee" 
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
    version="2.5">
  <display-name>Clojure Guestbook</display-name>
  <servlet>
    <servlet-name>gb</servlet-name>
    <servlet-class>guestbook.servlet</servlet-class>
  </servlet>
  <servlet-mapping>
    <servlet-name>gb</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
</web-app>

Create an appengine-web.xml, putting your application ID into the application element.

war/WEB-INF/appengine-web.xml

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
  <application>[your application ID]</application>
  <version>1</version>
  <static-files />
  <resource-files />
</appengine-web-app>

Create an ant build.xml file that compiles your application to the classes directory.

build.xml

<project name="guestbook-clj" basedir="." default="compile">
  ... here you'll need to define project.classpath ...
  ... see GitHub for the full working example file ...
  <target name="compile" depends="...">
    <java classname="clojure.lang.Compile" classpathref="project.classpath" failonerror="true">
      <classpath path="${src.dir}" />
      <sysproperty key="clojure.compile.path" value="${classes.dir}" />
      <arg value="guestbook.servlet" />
    </java>
  </target>
</project>

I've left out some of the details, but you can find a full working version on Github.

At this point you should be able to run your "Hello, World" iteration of the application locally using the development appserver and deploy to appspot using the appcfg executable (or an Ant task if you prefer). Now it's just a matter of building your application using the tools Google AppEngine, Clojure, and Compojure make available to you.

The User Service

AppEngine has a simple API for dealing with user accounts. Let's greet logged in users by name. You'll need to import com.google.appengine.api.users.UserServiceFactory.

(ns ...
  (:import
    (com.google.appengine.api.users UserServiceFactory)))

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html [:h1 "Hello, " (if user (.getNickname user) "World") "!"]))))

But we have to let users log in to see this work. The UserService also exposes methods for creating login and logout URLs.

...
  (GET "/"
    (let [user-service (UserServiceFactory/getUserService)
          user (.getCurrentUser user-service)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL user-service "/") "sign in")]
        [:p (link-to (.createLogoutURL user-service "/") "sign out")]))))

Now you should be able to log into your application and be greeted by name. On the dev appserver, the login page will let you provide any username and check a box to indicate whether you should be logged in as an administrator for your application. On the appspot servers, you'll get a proper-looking Google Accounts login page. The argument to createLoginURL and createLogoutURL is the path or URL the user should be redirected to after logging in or out.

I've extracted the basic user-lookup calls into a Ring middleware function and put it into the appengine-clj.users namespace in appengine-clj. Here's what our servlet looks like using that.

(ns guestbook.servlet
  ... you no longer need to import UserServiceFactory ...
  (:require
    [appengine-clj.users :as users]))

(defroutes guestbook-app
  (GET "/"
    (let [user-info (request :appengine-clj/user-info)
          user (user-info :user)]
      (html
        [:h1 "Hello, " (if user (.getNickname user) "World") "!"]
        [:p (link-to (.createLoginURL (user-info :user-service) "/") "sign in")]
        [:p (link-to (.createLogoutURL (user-info :user-service) "/") "sign out")]))))

(defservice (users/wrap-with-user-info guestbook-app))

It's about the same amount of code, it just looks a little more clojurey now. (I'm sorry: that's not a word.)

Datastore

Next let's collect guestbook entries and put them in the Datastore. AppEngine for Java has support for a couple of standard Java persistence APIs, JDO and JPA. But we'll use the lower-level datastore API, which seems a better fit for a dynamic language like Clojure (not to mention it doesn't require us to implement Java classes to persist).

The Java API for datastore is pretty simple, but conceptually it's different enough from a SQL database that it definitely takes some getting used to. (I for one am still figuring it out.)

Here's the sixty-second rundown of the bare essentials. The basic unit of persistence is the Entity, which has a Map of String-keyed properties. An Entity has a kind, which is a string denoting the type. (But keep in mind there's no schema here, so you can give any entity any properties.) An Entity is identified by a Key which is something more than a normal DBMS identifier because it can hold an Entity's association with a parent Entity. Besides using the Key, you can retrieve Entities with a Query, which searches either a single kind of entity or descendent entities of a single ancestor (or both, depending on which constructor you use) and can apply simple filtering and sorting.

The natural Clojurized form of an Entity seemed to be a map, so what I've started pulling out into appengine-clj.datastore is functions that allow Clojure code to work with an immutable map of keyword-keyed properties (plus :key and :kind) and have the library take care of translating into Entity objects. Currently there are just create and find methods, since that was all the basic guestbook needed. (But you know that Internet. I'll need a delete function before the week is out.)

Using appengine-clj.datastore, functions for creating and retrieving guestbook greetings are extremely simple.

(ns guestbook.greetings
  (:require [appengine-clj.datastore :as ds])
  (:import (com.google.appengine.api.datastore Query)))

(defn create [content author]
  (ds/create {:kind "Greeting" :author author :content content :date (java.util.Date.)}))

(defn find-all []
  (ds/find-all (doto (Query. "Greeting") (.addSort "date"))))

Note the creation of a com.google.appengine.api.datastore.Query (in find-all) that pulls back all Greetings and orders them by date. I've considered a couple of approaches for cleaning up creation of a query from Clojure, but I haven't decided between something that looks fairly idiomatic vs something that reads just like GQL. For the time being I'm sticking with the Java-interop style since even that is nicely terse and readable. Take a look at the tests for appengine-clj.datastore for more examples.

Speaking of which, this is a good time to mention that writing tests for datastore code is easy with appengine-clj.test-utils, which provides functions to set up an in-memory datastore. The dstest macro used there creates a fresh datastore for each test. If you're using a different testing framework or prefer different scoping, you can call ds-setup and ds-teardown yourself. (Do keep in mind that this is the development version of datastore, so we'll all need to keep an eye out for differences between that and the real Datastore service.)

HTML and Form Handling

Now that we've got our persistence straight (and tested), let's create a UI so users can sign the guestbook. At this point it's just plain Compojure code. We'll create one route to show the guestbook and a form to enter a greeting at "/" and another route for saving the greeting with a POST to "/sign".

Here's our function for signing the guestbook.

(defn sign-guestbook [params user]
  (greetings/create (params :content) (if user (.getNickname user)))
  (redirect-to "/"))

You can see there's very little to it. It takes the request parameters and a user, calls our greetings/create function, and redirects back to the guestbook.

The function for showing the guestbook is quite a bit more to swallow, since it includes our entire user interface.

(defn show-guestbook [{:keys [user user-service]}]
  (let [all-greetings (greetings/find-all)]
    (html [:html [:head [:title "Guestbook"]]
      [:body
        (if user
          [:p "Hello, " (.getNickname user) "! (You can "
            (link-to (.createLogoutURL user-service "/") "sign out")
            ".)"]
          [:p "Hello! (You can "
            (link-to (.createLoginURL user-service "/") "sign in")
            " to include your name with your greeting when you post.)"])
        (if (empty? all-greetings)
          [:p "The guestbook has no messages."]
          (map (fn [greeting]
            [:div
              [:p (if (greeting :author) [:strong (greeting :author)] "An anonymous guest") " wrote:"]
              [:blockquote (h (greeting :content))]])
            all-greetings))
        (form-to [POST "/sign"]
          [:div (text-area "content" "")]
          [:div (submit-button "Post Greeting")])]])))

It takes the user-info map, which it destructures to grab the user and UserService. It calls our greetings/find-all function to get the items to show and then uses Compojure's html helpers to create the document. For any real application you'd want to break the view down into smaller pieces to avoid such a huge nested chunk of vectors (or consider using another templating library like Enlive), but for this example I think it's easier to understand what's going on with the whole page in one function.

Finally here are the routes that wire it all together.

(defroutes guestbook-app
  (POST "/sign"
    (sign-guestbook params ((request :appengine-clj/user-info) :user)))
  (GET "/"
    (show-guestbook (request :appengine-clj/user-info))))

Here I'm using the :appengine-clj/user-info map that's been assoc'd to the request by the Ring middleware.

See the entire servlet file on GitHub, including some enhancements for styling and to see some other code to exercise Clojure features on AppEngine.

The Big Caveat

Two unusual aspects of the Google AppEngine environment create pretty major constraints on your ability to write idiomatic Clojure.

First, an AppEngine application runs in a security context that doesn't permit spawning threads, so you won't be able to use Agents, the clojure.parallel library, or Futures.

Second, one of the most exciting features of AppEngine is that your application will be deployed on Google's huge infrastructure, dynamically changing its footprint depending on demand. That means you'll potentially be running on many JVMs at once. Unfortunately this is a strange fit for Clojure's concurrency features, which are most useful when you have precise control over what lives on what JVM (and simplest when everything runs on one JVM). Since shared references (Vars, Refs, and Atoms) are shared only within a single JVM, they are not suitable for many of their typical uses when running on AppEngine. You should still use Clojure's atomic references (and their associated means of modification) for any state that it makes sense to keep global per-JVM, since there may be multiple threads serving requests in one JVM. But remember JVMs will come and go during the lifetime of your application, so anything truly global should go in the Datastore or Memcache.

More to Come

• I'll try and expand on this in the future with more write-ups, including a discussion of special handling for static files (which as of the version of the SDK I'm using works great on the appspot servers even with a /* servlet mapping but not on the local dev appserver, where servlet mappings win out over static files).
• If you'll sign my silly little guestbook on the appspot servers, I'd like to publish information on how many requests I got and how they performed.
• Google also provides Java APIs for caching, image manipulation, making HTTP requests, and email. I haven't even scratched the surface of those yet.
• With fresh support in AppEngine for scheduled tasks and upcoming support for task queues, there's more Clojure fun to be had.

Enjoy!

Update 7 September 2009: Late last week, the App Engine Team released version 1.2.5 of the SDK, including both XMPP (jabber instant messaging) support, which is brand new, and Task Queues, which had been available in the Python SDK but are now available for Java (and Clojure) applications.

clj-record Cleans Up Real Nice

Since clj-record started out as a sandbox for learning about Clojure macros, the project was a bit unstructured. Particularly weird was that the root source directory was the root of the project, so if you put that on your classpath, you also ended up with whatever random stuff was checked in at the top level on your classpath (for example TODO.txt and the script for running tests). That also gave me no good place to put dependencies, so you had to cobble together the right classpath on your own based on the instructions in the README.

I've now twisted the project into something more like a normal library layout. Source is rooted in /src. Dependencies are in /lib. Extra dependencies for running tests are in /test_lib. All this gets pulled together by Ant, which will run tests and spit out clj-record.jar (with ant all).

It was a painful moment starting work on that build.xml. Clojure would be a far better language for the build script—just about any language would be better than xml—but practicality won out: Clojure developers will have Ant installed. It was only a couple minutes before not having a real programming language there was an annoyance, and it will probably happen again. Hopefully once Programming Clojure is published, Stu Holloway will push forward with Lancet and we can all convert our build.xmls into build.cljs before long.

The Ant script also knows how to run a Nailgun server for VimClojure (with ant vim) that will have the source and tests on the classpath. (VimClojure is so badass.)

Certainly there's a lot to be said for the simplicity of just putting a thing on your classpath. You can still do that, just make it the project's /src directory instead of the root. At the moment the jar dependencies are checked into git, which could turn out to be a hassle for those who'd prefer to just put /src on their classpath. I could go to a separate download (as Compojure recently did) to keep the repository light. Let me know what you'd prefer.

Attribute Serialization for clj-record

I've just pushed clj-record support for attribute serialization to github. This uses clj-record callbacks and clojure.core's pr and read functions to allow you to store many standard Clojure and Java types as record attributes in varchar columns of your database.

I'm not crazy about calling this feature "serialization" since it's not using Java's serialization mechanism (though I'm very happy about Clojure's fairly human-readable pr/prn/read format). If you have thoughts on a better name, let me hear them.

Fancy Queries for clj-record

Thanks to another contribution from Tunde Ashafa, clj-record now has support for more advanced queries using functions in the new clj-record.query namespace.

Previously you could do this:

(manufacturer/find-records {:grade 98})
; or
(manufacturer/find-records {:name "Humedai Motors"})

Now you can do this:

; having required [clj-record.query :as q]
(manufacturer/find-records {:grade (q/greater-than 90)})
; or
(manufacturer/find-records {:name (q/like "Hume%")})

See the tests for more examples of what it does. See the source for how it's implemented. And please: recommend improvements.

clj-record support for MySQL and PostgreSQL

Thanks to pull requests from a couple of helpful github users (vishnu and tashafa), clj-record should now work with both MySQL and PostgreSQL.

The Ins and Outs of clj-record Validations

One of the first features I tackled in clj-record was a declarative validation API. Here I'll look at how it works, both in terms of how to use it and how it's implemented.

How to declare validations

The init-model form can include an option-group that opens with the keyword :validation like this.

(clj-record.core/init-model
  (:validation
    (:nickname "Nickname is required." #(not (empty? %)))))

Following :validation you provide any number of list forms with the attribute name as a keyword, the message that should be reported if the validation fails, and the function that implements the validation. It must take one argument, the value being validated, and return logical true or false. The function can be defined inline, as shown above, or a reference to a function defined elsewhere. (That is, even though init-model is a macro, your code will be evaluated as you expect.)

Built-in validations are provided in clj-record.validation.built-ins.

; assuming you've required [clj-record.validation.built-ins :as valid]
(clj-record.core/init-model
  (:validation
    (:nickname
      "Nickname must be three to fifteen letters."
      (valid/match #"^[A-Za-z]{3,15}$"))))

clj-record.validation.built-ins/match is a higher order function that takes a regular expression pattern and returns a validation function that passes only if the value matches the pattern (in whole or in part, so use ^ and $ in your pattern if you want to match the whole string).

Some of the functions in clj-record.validation.built-ins are, like match, higher order functions that return validation functions. Others, for example numeric?, are simple validation functions that you refer to directly.

(clj-record.core/init-model
  (:validation
    (:age "Age ain't nothing but a number." valid/numeric?)))

Since they're just normal functions, the built-ins can be combined just like any other functions.

(clj-record.core/init-model
  (:validation
    (:age "Age ain't nothing but a number (or nil)."
      #(or (nil? %) (valid/numeric? %)))))

How to run validations

ActiveRecord's validation API is nice looking in the normal case but conceptually a bit nasty. The valid? method looks like a pure predicate, but it mutates the model behind the scenes, clearing and re-populating its errors collection each time it's called. Since immutability is highly valued in functional languages, it was clear a similar approach wouldn't be appropriate for clj-record.

Model validations in clj-record are run using the validate function of the model namespace, which returns a validation-result. (If you can suggest a better name for that, please do.) The result can be inspected with the predicate clj-record.validation/valid? and messages for an attribute can be retrieved with clj-record.validation/messages-for, which takes the attribute name keyword and returns a collection of messages.

How validations work

The details of how validations are implemented shouldn't be relevant for normal use, but since clj-record is very young, it's pretty likely you'll need to look at the internals. If you're new to Clojure (or LISP macros in general) perhaps you'll also find this a useful case-study. I'm also interested in suggestions for improvements (to both the internals and the public-facing API).

First let's look at how option-groups in the init-model form work. The init-model macro uses the opening keyword of each option-group to look up a namespace. The validation option-group opens with :validation, so the namespace will be clj-record.validation.

It then calls a function in that namespace called expand-init-option once for each option-form in the option-group. (In the first example above, the only option-form is "(:nickname "Nickname is required." #(not (empty? %)))," but there can be more.) expand-init-option takes as arguments the model name (as a string) followed by whatever appeared in the option-form and returns a form that will appear in the expansion of init-model.

As you can guess from the examples above, clj-record.validation/expand-init-option takes as arguments the model-name followed by the (unevaluated) attribute name, message, and validation function. It returns a syntax-quoted form that calls clj-record.validation/add-validation with those same arguments verbatim, which in turn adds a validation to the (mutable) model metadata of your model.

So this:

; in a model namespace called com.example.model.foo
(clj-record.core/init-model
  (:validation
    (:nickname "Nickname is required." #(not (empty? %)))
    (:age "Age ain't nothing but a number." valid/numeric?)))

is equivalent to this:

; in a model namespace called com.example.model.foo
(clj-record.core/init-model)
(clj-record.validation/add-validation "foo"
  :nickname "Nickname is required." #(not (empty? %)))
(clj-record.validation/add-validation "foo"
  :age "Age ain't nothing but a number." valid/numeric?)

since the init-model macro will expand to a form including those exact add-validation calls.

Notice that very little work is happening in the macro-expansion. Keeping the macro layer very thin yields a number of benefits.

• First, it makes it trivial to allow references in the option-forms to work as you'd expect. In an earlier version of the implementation, model-metadata was built up at macro-expansion time, but it turned out I had to jump through extra eval hoops to get symbols to resolve to the right thing.
• Second, it keeps things working with AOT (ahead-of-time compilation). After jumping through the eval hoops, I decided to test a pre-compiled model class and realized that the in-memory model-metadata built up at macro-expansion time didn't exist, because macros had been expanded back when I compiled. Oops!
• Third, it makes testing much easier. I can define a funny validation for just one test and don't have to worry about keeping it passing in other tests. At the end of a test I just repoint a mutable ref back at the old value, and the model-metadata is back to its original shape. (I actually haven't yet used that technique for validation tests but will probably start soon. I do already use it in the tests for callbacks.)
• Finally, it reduces conceptual overhead. Macros are complicated. Anything you can do to make sure macro-expansion is just a simple conversion into something unmagical helps reduce head scratching.

Internally the validations are stored in model-metadata as a vector, since I wanted to run them in the order they're defined (for no real reason other than predictability). The validation result returned by validate is just a clojure hash map, but I provided valid? and messages-for so the implementation is abstracted. (Currently valid? is just empty? and messages-for is just get, but perhaps they'll get fancier later.)

So much for validations. Next week, maybe callbacks ... or maybe I'll actually do some coding.

clj-record: ActiveRecord for Clojure

I've spent a chunk of the last month (and an absurd amount of the last four days) working on clj-record: an ActiveRecord-like library written in the Clojure programming language. (Clojure is a LISP dialect with badass concurrency features for the JVM.) clj-record aims to provide an idiomatic "functional" API for persistence with many of the features we've come to appreciate from ActiveRecord, currently including validation, (somewhat) convenient associations, and (some) callbacks.

(Persistence requires side-effects, so it can't really be functional, but it can be idiomatic and leverage a great deal of power from being written in a functional language.)

Here's what a model definition looks like:

(ns clj-record.test.model.manufacturer
  (:require clj-record.boot))

(clj-record.core/init-model
  (:associations
    (has-many products))
  (:validation
    (:name "empty!" #(not (empty? %)))
    (:founded "must be numeric" #(or (nil? %) (not (re-find #"\D" %))))))

Coming soon:

• Even more convenient associations
• Even more callbacks
• Serialization of data structures as attribute values
• A fancy query API
• Other cool stuff

Check it out!

Show Me Your Meta

I'm working on a presentation on Ruby metaprogramming, and I'd love to have more examples from the outside world. If there's any piece of metaprogramming you've seen in open-source code that you thought was particularly clever, readable, convoluted, or gratuitous, please comment or email with links to the source (or at least a pointer in the general direction).

If you have some non-public-source example you think is interesting, please feel free to email me along with information about how I can use the example (and whether you want "credit" for supplying it).

Thanks!

ConstantizeAttribute to support Renum on Rails

The most obvious shortcoming of Renum is that there hasn't been any clean way to use enumerated values as ActiveRecord attribute values. I've finally fixed that.

There were a couple of Rails features that seemed like they might be helpful but turned out not to be.

Rails' built-in serialize macro class method uses YAML to store items, which is fine for arrays and hashes but hideous for anything else (if you ever look at your database directly). Though YAML serialization worked on Renum-created enumerated values, deserializing a value created a new instance. That instance might turn out to work fine for your needs, but it wouldn't actually be the instance it ought to be (i.e., the one the constant points to), so it might surprise and disappoint you in subtle ways. The other huge downside to serializing the instance (and its instance variables) is that a change in the encapsulated contents of the enumerated value could break things.

Rails' built-in composed_of macro class method could have been made to work, but it wouldn't have been pretty. The default handling would have required Renum to redefine new in generated enumeration classes to do a lookup instead of allocating and initializing an instance. The latest releases of Rails provide :constructor and :converter options that would have allowed me to avoid messing with new, but it still would've been ugly, not to mention requiring a very recent version of Rails.

What I finally realized is that the beauty of constants is that I didn't need to write a lookup mechanism: Ruby already does that. All I needed to do was store the name of the constant when writing the attribute and constantize it when reading to get the proper value. I also realized that approach is in no way tied to Renum: It would also allow classes and modules to be attribute values, which could be helpful if, for example, you have a module to mix in or a service class to call based on some reference data.

So I wrote a tiny little Rails plugin called ConstantizeAttribute that does this for you. This example pretty much says it all:

# ... your migration ...
create_table :robots do |t|
  t.column :behavior_module, :string
end

# ... your model ...
class Robot < ActiveRecord::Base
  constantize_attribute :behavior_module
end

# ... some classes or modules you want to store as attribute values ...
module RobotBehaviors
  module Handy
    def self.encounter
      "Is there anything I can help you with?"
    end
  end

  module Evil
    def self.encounter
      "I will destroy all humans."
    end
  end
end

# ... so now,
robby = Robot.create :behavior_module => RobotBehaviors::Evil

# Now "RobotBehaviors::Evil" is in the behavior_module column.

robby.behavior_module.encounter # => "I will destroy all humans."

robby.update_attribute :behavior_module, RobotBehaviors::Handy

# Now "RobotBehaviors::Handy" is in the behavior_module column.

robby = Robot.find :first

robby.behavior_module.encounter # => "Is there anything I can help you with?"

Install ConstantizeAttribute with

script/plugin install git://github.com/duelinmarkers/constantize_attribute.git

if your Rails is recent enough to install from git or grab a copy of the repo manually and drop the plugin in place. (There's no install script to worry about.)

The cool thing about this is that it works with old version of Renum without any changes, and it all happens in only about 15 lines of code.

A Ruby Metaprogramming Introduction Updated

Update: The Rspec conversion is done, and I added a section demonstrating macro methods. Read the latest here.

---

Early last year I published A Ruby Metaprogramming Introduction on the Practical Ruby blog. Earlier this year while commuting back and forth between Grand Central and White Plains—not a place you should strive to commute to, though MetroNorth is far better than the subway for giving you room to work on a laptop—I started making refinements and converting it to use Rspec instead of custom Test::Unit extensions.

I put it aside for a while but am now resolved to continue work on it. I've just put that work onto Github, so anyone with an interest can take a look, browse history, and even submit suggested patches.

The project is located here:
http://github.com/duelinmarkers/metaprogramming_introduction/tree

If you're annoyed at the half-Rspec state of things, you can read the originally published version in this tree. (It's way prettier as rendered on Github than I could probably manage to get it on any blog.)