Monday, December 15, 2008

webmachine in the wild

I recently discovered a couple of interesting places that webmachine is being used aside from our own applications:

Elisa Estonia are using webmachine as the web frontend to their SMS gateway.
(and have also provided useful contributions back to the webmachine source)

Kevin Smith is teaching a "Hands-On Erlang" class which includes a section on building a server app with webmachine.

Thursday, November 27, 2008

Thanks, Taavi!

Webmachine has received its first contribution from someone outside the core development team.

Taavi Talvik pointed out that some HTTP clients such as curl do not send a Content-Type by default when making PUT requests, and that it would be nicer to be able to work in such cases. He sent a patch with a suggested fix for this as well as a fix allowing PUT requests to more easily short-circuit to a desired response code.

Most of this submission is included in Webmachine 0.17. The only exception is that I did not add a default receiver for such PUT requests. This is out of a desire for explicitness in Webmachine resources. If you want to receive incoming PUT or POST requests, you must define what you will do with them.

Thank you, Taavi, for your contribution.

Wednesday, October 22, 2008

The BeeBole Erlang/Web Tutorial, Webmachine-Style

[EDIT: more up-to-date and very interesting versions of this app rewrite have been done by Bryan Fink (using a Riak backend) and by Serge (updating for Webmachine 1.x and implementing some of the improvements I suggested here). Either of those versions should be considered a better place to start than the code linked from this post.]

Hughes at BeeBole posted a neat screencast tutorial, building a small but fun Web application in mochiweb. It's worth watching.

Mochiweb is a fantastic programmable HTTP stack - but it's not a Web application toolkit. For that reason, when writing an app using only mochiweb one tends to do exactly what Hughes did in his tutorial: put several logical layers of functionality together in one big loop. You could argue "it's a tutorial on how to use mochiweb", implying that you might well benefit from having an amalgam of functionality (e.g., protocol, application code) exposed. While true, that also obscures important details and muddles the inherent value of separate logical layers that a good Web application embodies.

Webmachine, built on top of mochiweb, allows you to focus on writing your app while ensuring that things like caching behavior, content negotiation, and response codes, to name a few, are both purposeful and correct; think of it as an ambassador to HTTP: a lightweight, higher level framework for utilizing the excellent mechanics of mochiweb in ways that maximize your ability to focus on what you set out to accomplish: making a good Web application.

This all sounds nice, but -- to help me decide if it actually helped beginners in that way -- I tried a quick exercise of implementing Hughes' tutorial application as a Webmachine resource.

(note: The snippets here, concatenated, make up the entirety of the resource code. This really is all there is to it.)

I started by creating a Webmachine quick start directory and putting this dispatch line into the _sup file:
{dispatch, [{['*'], stickyNotes_webresource, [DocRoot]}]}]

This causes all requests to be sent to the stickyNotes webresource module, with their full path passed along and the DocRoot used as an initialization parameter to that resource. If I was doing this from scratch I'd have broken up the "notes" data and the static file delivery into separate resources, but the goal here is a simple work-alike. Now we need to write stickyNotes_webresource.erl:

-export([init/1, process_post/2, allowed_methods/2,
content_types_provided/2, resource_exists/2, provide_content/2]).
-record(context, {docroot,fullpath}).

The module and include_lib lines make up the sum of our boilerplate. The export declares which of the Webmachine resource functions we intend to override, and the record declares the data structure we'll use as our context threaded from function to function: a simple two-element struct.

init([DocRoot]) -> {ok, #context{docroot=DocRoot}}.

The Webmachine resource gets initialized for each request, and stows the docroot away for later use in serving up the static parts of the application.

allowed_methods(ReqProps, Context) ->
Path = ?PATH(ReqProps),
case Path of
"notes" ->
{['POST'], Context};
_ ->
{['HEAD', 'GET'], Context}

This declares that the URI /notes will only handle POST operations, and all other URIs served will only handle GET and HEAD operations. If a request is made against a URI with a different method from those allowed here, Webmachine will send a 405 Method Not Allowed response code.

That behavior is different from Hughes' tutorial code, which sends a 501 Not Implemented, regardless of URI, where the method is other than GET, HEAD, or POST. In that original tutorial, if a request goes to a working URI but with the wrong method (e.g. GET /notes) then a 404 Not Found is sent instead. Very confusing!

One of the core purposes of Webmachine is to encapsulate as much of the semantics of HTTP as possible so that applications don't have to recreate it. It's very easy for a smart person to get these details wrong, so there is value in not making them do it over and over again.

resource_exists(ReqProps, Context) ->
case ?PATH(ReqProps) of
"notes" -> {true, Context};
"" -> {true, Context#context{fullpath=
filename:join([Context#context.docroot, "index.html"])}};
Path ->
case mochiweb_util:safe_relative_path(Path) of
undefined -> {false, Context};
SafePath ->
FPath = filename:join([Context#context.docroot, SafePath]),
case filelib:is_regular(FPath) of
true -> {true, Context#context{fullpath=FPath}};
_ -> {false, Context}

This is conceptually simple (resource_exists simply decides whether to send a 404 or to proceed) but it ends up being the longest function because I decided to be explicit about checking the correctness of the URI path.

Also, this is where we set the fullpath element of our context object in the cases where we are going to serve a static file.

content_types_provided(ReqProps, Context) ->
Path = ?PATH(ReqProps),
case Path of
"" -> {[{"text/html", provide_content}], Context};
_ -> {[{webmachine_util:guess_mime(Path), provide_content}], Context}

provide_content(_ReqProps, Context) ->
{ok, Value} = file:read_file(Context#context.fullpath),
{Value, Context}.

These functions perform our content type negotiation and deliver the actual responses in the case of static file delivery. The first case entry ("") is there to account for the special index.html at the root.

process_post(ReqProps, Context) ->
Req = ?REQ(ReqProps),
Data = mochiweb_util:parse_qs(Req:recv_body()),
Struct = mochijson2:decode(proplists:get_value("json", Data)),
Act = list_to_atom(binary_to_list(struct:get_value(<<"action">>, Struct))),
Req:add_response_header("Content-Type", "application/json"),
{true, Context}.

This, the last part of our resource definition, handles the POST requests. All of these are already known (via allowed_methods) to be to the /notes URI path.

I would have done a few things differently than this if I was creating a tutorial from scratch. For starters, I would have done all of the JSON fetching ("read") in a GET operation instead of tunneling it through POST. I also would have made the POSTs directly in the media type application/json instead of burying the JSON inside a form-encoded body. Minor things like good HTTP behavior aside, it's a nice little demo and engaging enough to be a fun start for those new to Web programming in Erlang.

If this application was rewritten to have notes be proper addressable resources, it would then be trivial in the Webmachine variant to add the optimizations that were (sensibly) left out from the tutorial such as content-encoding selection or etags and other caching support.

Mochiweb is a great and powerful HTTP programming system. Webmachine, atop mochiweb, gives Web developers a higher-level platform to program against. This allows them to more easily develop general-purpose and extensible Web applications. The Webmachine source is currently available via SVN and soon to be made available in versioned tarballs. I've also made the Webmachine version of the tutorial code available in case anyone would like to play with it. If you download the tutorial and you already have a working Erlang/OTP installation, then you should be able to get running simply by:

cd /tmp (or wherever)
tar xzf webmachine-stickynotes.tar.gz
cd webmachine-stickynotes
(in the erlang shell, the very first time, run stickydb:reset().)
(point a browser at http://localhost:8000/)


Tuesday, September 23, 2008

Webmachine is a resource server for the Web.

I don't like web frameworks. They're mostly-complete Web applications with a few places you get to modify. If your biggest goal is to get something running instantly, and your something is shaped enough like an existing framework, then it might be your most effective way to get started. However, as soon as things inevitably get interesting, or you want to use a a different data storage system or some aspect of HTTP that your framework didn't envision... well, your handy framework is now something you have to spend time working around instead of with.

One of the first technical questions to deal with in a Web startup is "which Web framework should we use?" If the term wasn't already taken I would have imagined a Web framework to be something that gave you a place to define the representations and other Web-relevant properties of your application's resources -- with the emphasis that the first-class things on the Web are resources and that their essential properties of interaction are already quite well defined and usefully constrained. It was a Web framework about the Web that I really wanted, but I didn't see any.

I knew I wasn't the first one to ask. People like Bill have been wondering “where are the REST toolkits?” for a few years, but nothing really strong has yet emerged. Mark Nottingham has pointed out that even the better Web frameworks don't realize the power of having generic semantics. I agree, but I think that his examples didn't quite go far enough. Last December, Andy Gross and I realized that we could write something good enough for our own needs easily -- so we did.

Introducing Webmachine: a layer that adds HTTP semantic awareness on top of the excellent bit-pushing and HTTP syntax-management provided by mochiweb. Webmachine is not yet complete, and it's not perfect, but it has worked well for us and we'd like for there to be more conversation about what REST tooling should be like.

A Web application in Webmachine is a set of resources, and resources are defined by functions or predicates over their state. That might sound abstract or confusing, but it turns out to be quite simple to implement. I've provided some trivial examples to show just how simple it is.

The initial implementation of Webmachine was inspired in large part by version 1 of Alan Dean's HTTP flow diagram. I subsequently worked with Alan to improve the diagram as we developed Webmachine. Webmachine's request handling engine reflects that diagram explicitly.

If you decide to take it for a spin or read the source, please feel free to drop us a line.

A brief language digression: when we started writing a system in Erlang we assumed that we'd only be able to use it for some special-purpose backend work. We were wrong. It's been great for rapid development of interesting Web applications, and developers seem to like it. All of our server-side work, from web request handling to page templating to object storage has ended up being in Erlang not out of any dogma but because it has been such a practical tool.

At some point I hope to also post here about our choices in templating language, our unusual decisions for data storage, building practical systems around linked data, and how we've gained business value from representing our core data in multiple media types. But part of the point I hope to make with Webmachine today is that those choices ought to be separable from your choice of Web / REST tools.