I tend to view “web applications” as “HTTP interfaces” to underlying, independent functionality. So, you build an application and then expose that over HTTP to clients — either as browser application or an API end point.

This isn’t just a semantic shift — it drives the entire approach to building software!

For many developers, building software means building web applications. I’ve seen “web apps” used for everything from monitoring to batch processing. For them, every software problem is cast as “how do I solve this using my favorite web framework?”

This is the application view for web-bound developers:

This is the traditional application view:

Maybe this is a trivial difference. Who cares, right?

There are a lot of reasons why a systems view is the Right Way to approach software development. For one, a well designed application hosting environment (e.g. an operating system) encourages independence. Web frameworks generally don’t list “application independence” as a priority — and if they do, they probably leverage the operating system (e.g. fork-exec).

Independence in software is a topic for many, many blog posts :)

In this post, we’ll aspire only to use a simple tool to HTTP-enable an existing Erlang application!

modlib

modlib is a small library that makes it easy to build web applications that run in Erlang’s built-in http server.

What’s so special about Erlang’s http server?

• It comes with Erlang, so you can avoid a dependency
• It’s designed like Apache, using “mods” that can be configured as needed to quickly add features to your application
• It works well within the “systems” view — i.e. you can easily maintain independence across your application components

The big downside of Erlang’s http server is that it’s painful to work with.

modlib makes it easy.

The Application

We’re not going to start with a web application — we’re going to start with the underlying functionality. Once that’s in place, we’ll add an HTTP interface.

Let’s borrow some code from e2. From an OS shell:

$ git clone https://github.com/gar1t/e2
$ cp -a e2/examples/mydb modlib-example

Build the project:

$ cd modlib-example
$ make

This will compile the “mydb” example — a simple key/value store accessible using a TCP socket protocol. You can read all about this example in the e2 tutorial.

Start the mydb application in an Erlang shell:

$ make opts="-config test -s mydb" shell

This make target starts Erlang using the test.config file and starts the mydb application.

You can test the mydb data API in the Erlang shell (at each > prompt, type the text as listed below and press ENTER — you should see the corresponding output printed in the Erlang shell):

1> mydb_data:get("msg").
error
2> mydb_data:put("msg", "Hello mydb").
ok
3> mydb_data:get("msg").
{ok,"Hello mydb"}
4> mydb_data:del("msg").
ok
5> mydb_data:get("msg").
error

This is an example of an independently running application. The mydb database is running as an Erlang process — and we use the API defined in mydb_data to access it.

The database also handles TCP connections. Let’s see how this works.

From an OS shell, connect to the database using telnet:

$ telnet localhost 2222

To see how the TCP in action, type the commands below that follow the >> prompt (NOTE the >> prompt is not actually displayed in telnet — e.g. to execute the first command below, just type “GET msg” and then ENTER):

>> GET msg
-ERROR
>> PUT msg Hello mydb
+OK
>> GET msg
+Hello mydb
>> DEL msg
+OK
>> GET msg
-ERROR

This is an example of a TCP interface to the database. This interface supports multiple simultaneous connections, making mydb a concurrent key/value store!

Adding an HTTP Interface

We’ve just seen the “real” app. It works perfectly well on its own, but we also want to support client access over HTTP. This is where modlib comes in.

Let’s add modlib to as a build dependency. Edit rebar.config, located in the modlib example directory, so that it looks like this:

%% -*- erlang -*-
{erl_opts, [debug_info]}.
{deps,
 [{e2, ".*", {git, "git://github.com/gar1t/e2.git"}},
   {modlib, ".*", {git, "git://github.com/gar1t/modlib.git"}}]}.

Get the new dependency by running this from the OS shell:

$ make -B deps

mydb_http stub

I tend to use the naming convention that Erlang modules that handle HTTP requests end with “_http”.

Create the file src/mydb_http.erl that looks like this:

-module(mydb_http).

-include_lib("modlib/include/webapp.hrl").

-export([request/3]).

request("GET", Key, _Info) ->
    {ok, ["TODO: GET ", Key]};
request("PUT", Key, Info) ->
    {ok, ["TODO: PUT ", Key, " = ", modlib:body(Info)]};
request("DELETE", Key, _Info) ->
    {ok, ["TODO: DELETE ", Key]}.

This is a modlib “web app” — it’s a simple stub for an HTTP interface. It’s hard to imagine a simpler interface! But we’ll see in a moment that this is all we need.

Compile the new module by running this OS command in the mydb example root directory:

$ make

Configure modlib

Next we need to setup the HTTP server. This is done by defining a server in the modlib configuration. Modify test.config, located in the root mydb example directory, to look like this:

[{mydb, [{server_port, 2222}, {db_file, "test.db"}]},
 {modlib, [{servers, [{8080, [{modules, [mydb_http]}]}]}]}].

The modlib entry defined a single server running on port 8080. The server uses one “mod” — our new mydb_http web app!

Exit the Erlang shell by typing CTRL-C twice. Restart it using the same command:

$ make opts="-config test -s mydb" shell

This starts the mydb application as before: the same Erlang and TCP socket interfaces are available.

Let’s now start the modlib application. In the Erlang shell, run:

1> application:start(modlib).
ok

As before, type the Erlang statements following the “>” prompt and press ENTER.

When the modlib application starts, it starts the servers specified in the configuration file. In this case, an HTTP server is running on port 8080.

Use an HTTP client like curl to test the interface:

$ curl localhost:8080/foo
TODO: GET /foo
$ curl localhost:8080/foo -X PUT --data hello
TODO: PUT /foo = hello
$ curl localhost:8080/foo -X DELETE
TODO: DELETE /foo

Sweet!

Finish mydb_http

Modify mydb_http to look like this:

-module(mydb_http).

-include_lib("modlib/include/webapp.hrl").

-export([request/3]).

request("GET", Key, _Info) ->
    handle_get(mydb_data:get(Key));
request("PUT", Key, Info) ->
    handle_put(mydb_data:put(Key, modlib:body(Info)));
request("DELETE", Key, _Info) ->
    handle_del(mydb_data:del(Key)).

handle_get({ok, Value}) -> {ok, Value};
handle_get(error) -> {not_found, "Not Found"}.

handle_put(ok) -> {ok, ""}.

handle_del(ok) -> {ok, ""}.

Believe it or not, this is a complete HTTP interface — 13 lines of code!

Compile the changes:

$ make

The example is setup to automatically reload the changed module — you don’t need to restart the Erlang shell for the change to take effect.

Use your HTTP client (e.g. curl) to test your new interface!

Wrapping Up

If you like the idea of tacking on HTTP interfaces to your apps, as needed, modlib is a drop dead easy way to do it.

But don’t let this ultra-simple approach fool you! You can build very advanced web applications using what you’ve seen here, plus a little extra, depending on what you need:

• mod_get to serve static content from a directory
• mod_auth to add HTTP authentication
• ErlyDTL
• Various Erlang database bindings, or perhaps boss_db if you like ORMs
• A great JavaScript library like YUI

You can get from “simple” to “advanced” using small steps. Add only what you need when you need it!

A Note On Middleware

The “mods” design of the Erlang http server uses a middleware pattern: you can insert software components into the middle of a client/server exchange.

This is a Very Good Pattern.

Python’s rich web ecosystem owes its success to an effective middleware strategy in WSGI.

Erlang needs this.

The “mod” interface defined by httpd is an example, though a terrible one — it’s is blindingly complicated. modlib could serve as a step toward improving it, though it was never envisioned as a generalized HTTP interface ala WSGI.

To achieve WSGI like success in Erlang, the Erlang community needs a drop-dead simple interface that can be used to chain HTTP handlers together. If this was done right and promoted effectively, I think we’d see Erlang emerge as a top tier platform for “web development”.

References

• modlib
• Erlang HTTP server

Here my presentation of e2 at Erlang Factory in San Francisco:

Presentation Slides for e2

This is a Landslide presentation. In keeping with my new found plain text fetish, Landslide lets you author presentations in Markdown and then generate a reasonably nice HTML/JavaScript presentation.

In theory, it lets you generate a PDF, but I wasn’t able to get images to show up.

In general I found Landslide to be very rough (bugs aplenty) but the promise is there: author content in simple, text formats and use it to generate alternative formats.

Blogofile is blog software that lets you write blog posts as plain text files, which are compiled into a web site.

It’s enlightened, I think.

It’s a hell of a lot easier to get going than WordPress. But far more importantly, it treats your blog as source code. I’d like more things in life to be treated like code.

Background

A week ago I ran an Arch system upgrade that hosed sshd. So I had to rebuild the server. In EC2 that’s pretty easy if you’re using EBS.

But no one likes that sort of hassle — here’s a trace of my emotional state changes:

1. Fucking hell Arch, you can’t possibly be serious
2. Let’s just get this over with
3. I’ll only rebuild what’s absolutely necessary
4. Man is it easy to install software with Arch
5. There’s no way I’m dealing with WordPress again — I’ll find something simpler

Googled “file based blog” — not much help, but on closer inspection, something useful!

After a relatively painless experience of getting the required dependencies installed, I was running a stand-alone blog that served content from plain ol’ text files that sat in a plain ol’ file system. I used plain ol’ Emacs and ran a build. A little Nginx proxy_pass love and I had a sane, cogent blog!

What’s Wrong with WordPress?

Nothing, it’s a terrific piece of software. When I used it, I was continually blow away by it’s polish, feature set, and relative ease of use.

But a couple things caused my eye to wander:

• While it’s easy to setup relative to a lot of crap out there, WordPress is not that easy to setup. I wanted something that I could install as a single Arch package and run as a daemon… simple!

• I absolutely hate storing text content in a database. I absolutely hate editing text content in a web based WYSIWYG editor. I want text files. There are lots of tools I can use with text files.

I wanted a system where I could open my laptop, hack away on content and then publish that when I was ready. This is a coding problem. I don’t need any other memes. I just want code and a build process. I know this will work.

Text files are the lingua franca of information!

Missing WordPress, a Little

Okay, WordPress did give me a gorgeous, modern, ultra hip blog with just the click of a few buttons.

So, I need to dig up some templates, CSS, etc. and make this work myself. That’s not the end of the world — but it does raise the barrier to adopt this approach.

That said, it’s an easy trade of to make: my “blog” now is a self-contained directory of text files stored in github! To restore it, I use a git clone operation. I have a complete history of all changes. If I want to make my life complicated, I can work on branches for previewing changes and then merge them into “master”.

Whatever, the point is it’s code.

Much Potential With Blogfile

Blogfile is really a web site generator that uses Python, Mako templates and various markup flavors to generate content. It’s a content centric tool that sits somewhere between a CMS and raw HTML.

To generate content, you register a controller with Blogfile. The “blog” is one type of controller. But it looks pretty easy to create new types.

So I see myself using Blogfile for other content management, assuming this experiment goes well. In particular, I’d like a place to store my presentation slides. E.g. use markdown to organize a “slide” that gets turned into HTML that gets inserted into an S5 presentation.

Ideas:

• Presentation generator (S5)
• List of YouTube videos
• Project web sites (e.g. e2)

Random Comments

Blogofile follows Sphinx’s example.

This Blogofile project seems to have slowed or even stalled.

blogofile.com site is built with Blogofile, but it doesn’t make systematic use of markdown for content. E.g. non blog posts are written in HTML, embedded in Mako templates.

Using Disqus to enable comments on a static site is a great example of separation of concerns and is brilliant! Now if there was only some way to do this for ad syndication, hmmm…