Wednesday, October 22, 2008

The BeeBole Erlang/Web Tutorial, Webmachine-Style

[EDIT: more up-to-date and very interesting versions of this app rewrite have been done by Bryan Fink (using a Riak backend) and by Serge (updating for Webmachine 1.x and implementing some of the improvements I suggested here). Either of those versions should be considered a better place to start than the code linked from this post.]

Hughes at BeeBole posted a neat screencast tutorial, building a small but fun Web application in mochiweb. It's worth watching.

Mochiweb is a fantastic programmable HTTP stack - but it's not a Web application toolkit. For that reason, when writing an app using only mochiweb one tends to do exactly what Hughes did in his tutorial: put several logical layers of functionality together in one big loop. You could argue "it's a tutorial on how to use mochiweb", implying that you might well benefit from having an amalgam of functionality (e.g., protocol, application code) exposed. While true, that also obscures important details and muddles the inherent value of separate logical layers that a good Web application embodies.

Webmachine, built on top of mochiweb, allows you to focus on writing your app while ensuring that things like caching behavior, content negotiation, and response codes, to name a few, are both purposeful and correct; think of it as an ambassador to HTTP: a lightweight, higher level framework for utilizing the excellent mechanics of mochiweb in ways that maximize your ability to focus on what you set out to accomplish: making a good Web application.

This all sounds nice, but -- to help me decide if it actually helped beginners in that way -- I tried a quick exercise of implementing Hughes' tutorial application as a Webmachine resource.

(note: The snippets here, concatenated, make up the entirety of the resource code. This really is all there is to it.)

I started by creating a Webmachine quick start directory and putting this dispatch line into the _sup file:
{dispatch, [{['*'], stickyNotes_webresource, [DocRoot]}]}]

This causes all requests to be sent to the stickyNotes webresource module, with their full path passed along and the DocRoot used as an initialization parameter to that resource. If I was doing this from scratch I'd have broken up the "notes" data and the static file delivery into separate resources, but the goal here is a simple work-alike. Now we need to write stickyNotes_webresource.erl:

-export([init/1, process_post/2, allowed_methods/2,
content_types_provided/2, resource_exists/2, provide_content/2]).
-record(context, {docroot,fullpath}).

The module and include_lib lines make up the sum of our boilerplate. The export declares which of the Webmachine resource functions we intend to override, and the record declares the data structure we'll use as our context threaded from function to function: a simple two-element struct.

init([DocRoot]) -> {ok, #context{docroot=DocRoot}}.

The Webmachine resource gets initialized for each request, and stows the docroot away for later use in serving up the static parts of the application.

allowed_methods(ReqProps, Context) ->
Path = ?PATH(ReqProps),
case Path of
"notes" ->
{['POST'], Context};
_ ->
{['HEAD', 'GET'], Context}

This declares that the URI /notes will only handle POST operations, and all other URIs served will only handle GET and HEAD operations. If a request is made against a URI with a different method from those allowed here, Webmachine will send a 405 Method Not Allowed response code.

That behavior is different from Hughes' tutorial code, which sends a 501 Not Implemented, regardless of URI, where the method is other than GET, HEAD, or POST. In that original tutorial, if a request goes to a working URI but with the wrong method (e.g. GET /notes) then a 404 Not Found is sent instead. Very confusing!

One of the core purposes of Webmachine is to encapsulate as much of the semantics of HTTP as possible so that applications don't have to recreate it. It's very easy for a smart person to get these details wrong, so there is value in not making them do it over and over again.

resource_exists(ReqProps, Context) ->
case ?PATH(ReqProps) of
"notes" -> {true, Context};
"" -> {true, Context#context{fullpath=
filename:join([Context#context.docroot, "index.html"])}};
Path ->
case mochiweb_util:safe_relative_path(Path) of
undefined -> {false, Context};
SafePath ->
FPath = filename:join([Context#context.docroot, SafePath]),
case filelib:is_regular(FPath) of
true -> {true, Context#context{fullpath=FPath}};
_ -> {false, Context}

This is conceptually simple (resource_exists simply decides whether to send a 404 or to proceed) but it ends up being the longest function because I decided to be explicit about checking the correctness of the URI path.

Also, this is where we set the fullpath element of our context object in the cases where we are going to serve a static file.

content_types_provided(ReqProps, Context) ->
Path = ?PATH(ReqProps),
case Path of
"" -> {[{"text/html", provide_content}], Context};
_ -> {[{webmachine_util:guess_mime(Path), provide_content}], Context}

provide_content(_ReqProps, Context) ->
{ok, Value} = file:read_file(Context#context.fullpath),
{Value, Context}.

These functions perform our content type negotiation and deliver the actual responses in the case of static file delivery. The first case entry ("") is there to account for the special index.html at the root.

process_post(ReqProps, Context) ->
Req = ?REQ(ReqProps),
Data = mochiweb_util:parse_qs(Req:recv_body()),
Struct = mochijson2:decode(proplists:get_value("json", Data)),
Act = list_to_atom(binary_to_list(struct:get_value(<<"action">>, Struct))),
Req:add_response_header("Content-Type", "application/json"),
{true, Context}.

This, the last part of our resource definition, handles the POST requests. All of these are already known (via allowed_methods) to be to the /notes URI path.

I would have done a few things differently than this if I was creating a tutorial from scratch. For starters, I would have done all of the JSON fetching ("read") in a GET operation instead of tunneling it through POST. I also would have made the POSTs directly in the media type application/json instead of burying the JSON inside a form-encoded body. Minor things like good HTTP behavior aside, it's a nice little demo and engaging enough to be a fun start for those new to Web programming in Erlang.

If this application was rewritten to have notes be proper addressable resources, it would then be trivial in the Webmachine variant to add the optimizations that were (sensibly) left out from the tutorial such as content-encoding selection or etags and other caching support.

Mochiweb is a great and powerful HTTP programming system. Webmachine, atop mochiweb, gives Web developers a higher-level platform to program against. This allows them to more easily develop general-purpose and extensible Web applications. The Webmachine source is currently available via SVN and soon to be made available in versioned tarballs. I've also made the Webmachine version of the tutorial code available in case anyone would like to play with it. If you download the tutorial and you already have a working Erlang/OTP installation, then you should be able to get running simply by:

cd /tmp (or wherever)
tar xzf webmachine-stickynotes.tar.gz
cd webmachine-stickynotes
(in the erlang shell, the very first time, run stickydb:reset().)
(point a browser at http://localhost:8000/)



Unknown said...

Really interesting article. I like the lightweight approach you have taken.

My tutorial is definitively not full fledge app minded as I wanted it as close to the mochiweb metal as possible ;)

Having said that, we are addressing some of the details you mentioned in your post but using an home made layer since we are not building a http general purpose application.

We are actually tunneling all of the 'Jsons' trough both the GET(jsonp) and the POST method and we are using a unique interface to handle them (the exceptions are few).
Keep up the good work.

John Wright said...

Hi Justin,

This is great stuff. As a small company just getting started with Mochiweb this will really help. I was looking for a super fast REST framework and thought I had found it with Mochiweb but this goes even further! I was wondering if you might give some more insight into the data analysis pieces you have built with Erlang if it's not too much secret sauce. I know you have released some of the components with distributerl but I was looking for insight into how you all answer arbitrary data analysis questions with your homegrown backend.

Justin Sheehy said...

John, I'm glad to hear that Webmachine might be a good fit for you. There will be another release fairly soon with some bugfixes and improvements. Please feel free to send me email whenever you wish, especially if you're using it to build anything new.

We're not prepared at this time to release any of our data analysis code beyond some basic building blocks like those in distributerl. There will certainly be more along those lines released later -- it will be used commercially and select portions will continue to be released as open source.


Mathias Picker said...

Really nice tutorial. I just found webmachine yesterday and I think it's just what I've been looking for. (Now for the frontend selection ;)

I generally prefer the continuation based aproach (seaside et al.) but since there are no continuations in erlang; I'll go the ajax head route, and for this webmachine is the perfect backend!

Cheers, Mathias

Mathias Picker said...

Combining webmachine with erlydtl gives me allmost all I need for speedy web development. It's remarkable easy to combine, I just had my first page up served from webmachine showing an earlydtl template in an hour or so; with no experience with both libraries (although knowing the django template syntax ;)

Thanks, and keep up the good work.

iWantToKeepAnon said...

I am really liking webmachine and the wmtrace facility (add by Bryan right?) is so helpful.

Thanks for the stickyNotes conversion. Maybe it's time to update it to webmachine 1.0? Although with the wiki guide it was not hard for me to upgrade it myself ... helps learn the framework too I guess.

The only thing I find missing is the ensurance that mnesia is started. I don't find the equivalent of 'stickyNotes:start()' in the webmachine version. So I either start it or reset it from the erlang shell, but that isn't the preferred way of doing it ... is it?

Justin Sheehy said...


stickyNotes:start() is present in the Webmachine version as well.

(as the same named function, in fact)

Look at the start/0 function in src/stickyNotes.erl in the tarball.

I'm unlikely to update it to Webmachine 1.x anytime soon, simply because it was a one-off example and I'd rather get some new things done.

Web Development Company said...

An interesting and informative article...

Thanks for the post.