Webmachine 1.3: streamed bodies, multipart forms, and efficiency

Easily the most requested feature for Webmachine since its release has been the ability to "stream" the request and/or response bodies, instead of having to receive or send them in one potentially-large hunk. As of the most recent version, this feature is now available. See the wiki page for details on the API.

A number of other changes are also in, such as multipart form parsing, improved efficiency by changing a gen_server (per request) into a parameterized module, and so on... but I suspect that the streamed bodies are what people are really looking for most. Enjoy!

REST and HTTP services as a business advantage

The advantages of HTTP as an application protocol (not just a transport) as opposed to many other networked service models are not abstract, idealized technical advantages. They directly affect your -- and your partners' -- cost of doing business.

At Basho, our services integrate out of necessity with those of many kinds of partner companies, including CRM, Business Intelligence, Search, and more. We consider ourselves lucky in general when a company we'd like to partner with exposes any consistent and documented interface for this purpose.

However, when those interfaces are SOAP or another RPC-shaped system it means that each integration is a fairly major new project even when the resulting connections between applications are conceptually small. This is because you have to learn the programming model of that other service and work as though you were a developer of that service -- learning their calling conventions, naming schemes, error conditions, and so on.

We recently had the pleasure of integrating with Jigsaw's data service. While they don't quite match up to the ideals of REST just yet, their service is young and the interface is already far better than that of many other business-to-business integration APIs. Not only did they deliver a cleaner and easier service than expected, I suspect that they did so at lower cost than many others. How?

By using HTTP.

Even the coarsest approximation of the Web's uniform interface gives you a much better running start than is possible with, say WSDL and SOAP. Jigsaw's Web interface isn't perfect (GET requests are idempotent but not safe, and a couple of status codes are incorrectly used) but it is simple and it isn't surprising. The fact that there is already a completely interoperable HTTP client in every major programming language means that, instead of using some WSDL to generate 10,000 lines of code to then put a client on, we were able to just jump in and immediately write working client code. The resulting client code was also about 20% as long as the manually written portion of our client code in comparable services that use SOAP.

I'm not talking about ideal systems, and I'm not talking about idealistic academic goals. I'm just talking about the simple realities of how your technical choices affect the level of effort that your partners must apply in order to work with you. That simple reality has a direct and powerful effect on the bottom line.

Video Slideshow, Introducing Webmachine

The Webmachine talk at Bay Area Erlang Factory 2009 went quite well. I received useful feedback, and some very interesting and productive conversations spun off after the talk.

For anyone interested who wasn't there, I have recorded a voiceover with the slides and made that video available here. The slides are the same ones used at the conference, but I trimmed the speaking portion a bit. This version is a bit under half an hour; it leaves out a few minor topics but still covers all of the material needed to introduce Webmachine.

Enjoy!

Webmachine 1.2

There are a few changes in webmachine-1.2 that deserve mention.

We simplified the API to the dispatcher module so that it can be used easily in a standalone fashion. In cases where another application (such as CouchDB) wants to use Webmachine-style dispatching, it is now easy to just call webmachine_dispatcher:dispatch/2 and get a useful result without any of the rest of Webmachine running. A trivial example:

1> webmachine_dispatcher:dispatch("/a",[{["a"],some_resource,[]}])
{some_resource,[],[],[],".",[]}

The other change that is most interesting from a feature point of view is that the request body is not read off the socket until the first time wrq:req_body/1 is called. This means that a resource can (for example) return an error response code without having to wait for the body to be pulled off the wire first.

There is also a change in the new_webmachine project creation script. Your list of dispatch terms will now by default be in a separate file ("priv/dispatch.conf") instead of directly in your application's _sup file.

This version is identified with the "webmachine-1.2" mercurial tag,

In upcoming versions, we hope to add a few much-clamored-for features such as host-based dispatching and incremental request/response body reading and writing.

A Simple Webmachine Example

Bryan Fink (of BeerRiot fame and a colleague at Basho) recently posted a great example of how easy it is to make a useful and working Webmachine resource.

He then followed up with more examples, showing how easy it is to add support for PUT, for authorization, and for
entity tags.

Yesterday's post wrapped up his short series by not only adding DELETE support but also reflecting on the nature of Webmachine and how it lets you improve the way you think about and use the power of HTTP in your applications.

Bryan knows both Erlang and Web programming well; his examples are worth the read.

Webmachine One Point Oh!

I am happy to announce the release of Webmachine 1.0.

In its short public life so far, Webmachine has already been used to build a range of Web applications, including a sales productivity tool, an SMS gateway, an HTTP caching intermediary, a content management system, the front end to a decentralized key/value data store, and more. It has been used as a central element in Erlang training courses, enabling students to write working, Web-friendly applications after only a day or two of exposure to the language. Some of its users have been delivering customer-facing Webmachine applications for nearly a year and a half now.

However, the 1.0 release isn't just an acknowledgment of that stability. It also introduces two major changes that are very beneficial to developers. One of these is a new debugging tool that is unlike anything we've seen elsewhere, allowing you to visualize in great detail how your Web resources process requests. Bryan Fink did most of the work on the debugging and tracing tool, and is the best person to explain how it works.

The other major change is to the developer API for resources, helping to make your resource functions to be referentially transparent. In case that phrase is new to you, it really means something quite simple. Resource functions no longer interact with a Req object that manipulates the response via side effects and stored state. Instead, the behavior is defined purely in terms of its input parameters and return value. For any given input a given function ought to return the same output and the side effects will be insignificant from the point of view of Webmachine's execution. This might not sound like a huge deal, but the difference it makes in terms of testability and re-use is huge.

Unit testability is vastly improved because the functions are fully isolated. You can write test inputs for your (e.g.) is_authorized function (and verify that it only returns true in the cases you want it to) without having those tests have to interact with anything else in the application. There's no need for anything resembling a "mock object" or other such silliness; when the inputs and outputs are simple records as opposed to gen_servers with side effects, you can just construct whole records for your test inputs, and inspect the output records for checking test results.

Referential transparency also enables much more in the way of static analysis. One item on the roadmap for a future version of Webmachine is a type analysis tool that can verify at compile time that the type signatures of your resource functions can only lead to valid HTTP behavior.

It's also worth noting that the backward incompatibility is well-contained and easy to get past. Our production application was switched entirely to the new API by one person in a day, and Bryan also converted all of BeerRiot in under two hours. A handy guide to upgrading shows just how simple it is.

As of the 1.0 release we're also moving Webmachine from Google Code to Bitbucket. The main reason for this is that we vastly prefer Mercurial to Subversion, but a number of other features on Bitbucket have also turned out to be nice. We're leaving the old version up on Google Code for a while, so that people with running applications can access both the old and new versions.

Enjoy!

Webmachine at Erlang Factory

I will be speaking at Erlang Factory Bay Area 09 on Webmachine.

I'm looking forward to the conference; it seems like there will be a very interesting crowd.