Monday, June 15, 2009

Webmachine 1.3: streamed bodies, multipart forms, and efficiency

Easily the most requested feature for Webmachine since its release has been the ability to "stream" the request and/or response bodies, instead of having to receive or send them in one potentially-large hunk. As of the most recent version, this feature is now available. See the wiki page for details on the API.

A number of other changes are also in, such as multipart form parsing, improved efficiency by changing a gen_server (per request) into a parameterized module, and so on... but I suspect that the streamed bodies are what people are really looking for most. Enjoy!

Monday, June 1, 2009

REST and HTTP services as a business advantage

The advantages of HTTP as an application protocol (not just a transport) as opposed to many other networked service models are not abstract, idealized technical advantages. They directly affect your -- and your partners' -- cost of doing business.

At Basho, our services integrate out of necessity with those of many kinds of partner companies, including CRM, Business Intelligence, Search, and more. We consider ourselves lucky in general when a company we'd like to partner with exposes any consistent and documented interface for this purpose.

However, when those interfaces are SOAP or another RPC-shaped system it means that each integration is a fairly major new project even when the resulting connections between applications are conceptually small. This is because you have to learn the programming model of that other service and work as though you were a developer of that service -- learning their calling conventions, naming schemes, error conditions, and so on.

We recently had the pleasure of integrating with Jigsaw's data service. While they don't quite match up to the ideals of REST just yet, their service is young and the interface is already far better than that of many other business-to-business integration APIs. Not only did they deliver a cleaner and easier service than expected, I suspect that they did so at lower cost than many others. How?

By using HTTP.

Even the coarsest approximation of the Web's uniform interface gives you a much better running start than is possible with, say WSDL and SOAP. Jigsaw's Web interface isn't perfect (GET requests are idempotent but not safe, and a couple of status codes are incorrectly used) but it is simple and it isn't surprising. The fact that there is already a completely interoperable HTTP client in every major programming language means that, instead of using some WSDL to generate 10,000 lines of code to then put a client on, we were able to just jump in and immediately write working client code. The resulting client code was also about 20% as long as the manually written portion of our client code in comparable services that use SOAP.

I'm not talking about ideal systems, and I'm not talking about idealistic academic goals. I'm just talking about the simple realities of how your technical choices affect the level of effort that your partners must apply in order to work with you. That simple reality has a direct and powerful effect on the bottom line.