Bill Clementson's Blog

Distel = Erlang-like Concurrency in Emacs

Thursday, July 19, 2007

In an earlier post (titled "Distel = Emacs erlang-mode++"), I talked about how Distel provides an enhanced Erlang Mode for Emacs. However, the Erlang Mode parts of Distel represent just an "application" of Distel (albeit, the most understood and widely-used application). In reality though, Distel is much more than "just" an enhanced Emacs mode for working with Erlang code. It is the embodiment (and excellent example) of a certain Erlang interoperability mechanism. To understand what I'm talking about, I'll provide a bit of background.

Erlang is a fantastic language for developing concurrent programs. However, sometimes you need to interact with non-Erlang programs as well. The Erlang Reference Manual and Interoperability Tutorial provide good information on the primary interoperability mechanisms that are built into Erlang - I'll summarize the comments on the two main interoperability mechanisms here:

1. Distributed Erlang: The Interoperability Tutorial states:

   "An Erlang runtime system is made into a distributed Erlang node by giving it a name. A distributed Erlang node can connect to and monitor other nodes, it is also possible to spawn processes at other nodes. Message passing and error handling between processes at different nodes are transparent. There exists a number of useful stdlib modules intended for use in a distributed Erlang system; for example, global which provides global name registration. The distribution mechanism is implemented using TCP/IP sockets.
   
   When to use: Distributed Erlang is primarily used for communication Erlang-Erlang. It can also be used for communication between Erlang and C, if the C program is implemented as a C node, see below.
   
   Where to read more: Distributed Erlang and some distributed programming techniques are described in the Erlang book. In the Erlang/OTP documentation there is a chapter about distributed Erlang in "Getting Started" (User's Guide). Relevent man pages are erlang (describes the BIFs) and global, net_adm, pg2, rpc, pool and slave."

   Although the above states that this mechanism "can also be used for communication between Erlang and C, if the C program is implemented as a C node", it should really state that this mechanism can be used for communication between Erlang and any other language if the language's program is implemented as a node.


2. Ports and Linked-In Drivers: The Interoperability Tutorial states:

   "Ports provide the basic mechanism for communication with the external world, from Erlang's point of view. They provide a byte-oriented interface to an external program. When a port has been created, Erlang can communicate with it by sending and receiving lists of bytes (not Erlang terms). This means that the programmer may have to invent a suitable encoding and decoding scheme.
   
   The actual implementation of the port mechanism depends on the platform. In the Unix case, pipes are used and the external program should as default read from standard input and write to standard output. Theoratically, the external program could be written in any programming language as long as it can handle the interprocess communication mechanism with which the port is implemented.
   
   The external program resides in another OS process than the Erlang runtime system. In some cases this is not acceptable, consider for example drivers with very hard time requirements. It is therefore possible to write a program in C according to certain principles and dynamically link it to the Erlang runtime system, this is called a linked-in driver.
   
   When to use: Being the basic mechanism, ports can be used for all kinds of interoperability situations where the Erlang program and the other program runs on the same machine. Programming is fairly straight-forward. Linked-in drivers involves writing certain callback functions in C. Very good skills are required as the code is linked to the Erlang runtime system.
   
   Warning! An erronous linked-in driver will cause the entire Erlang runtime system to leak memory, hang or crash.
   
   Where to read more: Ports are described in the "Miscellaneous Items" chapter of the Erlang book. Linked-in drivers are described in Appendix E. The BIF open_port/2 is documented in the man page for erlang. For linked-in drivers, the programmer needs to read the information in the man page for erl_ddll."

1. Separate standard Erlang and Elisp for a non-concurrent solution.
2. Separate concurrent solutions in both Erlang and Elisp/Distel (with the Elisp/Distel solution NOT interoperating with Erlang)
3. An interoperable Erlang/Elisp/Distel solution (with the Elisp/Distel node communicating with the Erlang node)

1. Separate standard Erlang and Elisp for a non-concurrent solution:


So, to start off, here's the code for a standard Erlang version of the beer song:

-module(beersong1).
-export([sing/0]).

template(0)  -> "~s of beer on the wall, ~s of beer.~nGo to the store and buy some more, 99 bottles of beer on the wall.~n";
template(_N) -> "~s of beer on the wall, ~s of beer.~nTake one down and pass it around, ~s of beer on the wall.~n~n".

create_verse(0)      -> io:format(template(0), phrase(0));
create_verse(Bottle) -> 
    io:format(template(Bottle), phrase(Bottle)),
    create_verse(Bottle-1).

phrase(0)      -> ["No more bottles", "no more bottles"];
phrase(1)      -> ["1 bottle", "1 bottle", "no more bottles"];
phrase(2)      -> ["2 bottles", "2 bottles", "1 bottle"];
phrase(Bottle) -> lists:duplicate(2, integer_to_list(Bottle) ++ " bottles") ++ [integer_to_list(Bottle-1) ++ " bottles"].

sing() -> create_verse(99).

and the Emacs Lisp equivalent code:

(defun template (n)
  (if (= n 0)
      "%s of beer on the wall, %s of beer.nGo to the store and buy some more, 99 bottles of beer on the wall.n"
    "%s of beer on the wall, %s of beer.nTake one down and pass it around, %s of beer on the wall.nn"))

(defun create-verse (bottle)
  (let ((phrases (phrase bottle)))
    (insert (format (template bottle) (first phrases) (second phrases) (third phrases))))
  (if (> bottle 0)
      (create-verse (1- bottle))))

(defun phrase (bottle)
  (case bottle
    (0 (list "No more bottles" "no more bottles"))
    (1 (list "1 bottle" "1 bottle" "no more bottles"))
    (2 (list "2 bottles" "2 bottles" "1 bottle"))
    (t (list (format "%s bottles" bottle) (format "%s bottles" bottle) (format "%s bottles" (1- bottle))))))

(defun sing ()
  (interactive)
  (create-verse 99))

Nothing special here - I just provided this code as a base example that is easy to understand and build on. When you run the Elisp code (using "M-x sing"), the output will be printed in whatever is the current buffer, so you might want to switch to an appropriate buffer first (e.g. - "*scratch*").


2. Separate concurrent solutions in both Erlang and Elisp/Distel (with the Elisp/Distel solution NOT interoperating with Erlang):


Here's a concurrent version in Erlang (I wrote about this one in an earlier post):

-module(beersong).
-export([sing/0]).

template(0)  -> "~s of beer on the wall, ~s of beer.~nGo to the store and buy some more, 99 bottles of beer on the wall.~n";
template(_N) -> "~s of beer on the wall, ~s of beer.~nTake one down and pass it around, ~s of beer on the wall.~n~n".

create_verse(0)      -> {0, io_lib:format(template(0), phrase(0))};
create_verse(Bottle) -> {Bottle, io_lib:format(template(Bottle), phrase(Bottle))}.

phrase(0)      -> ["No more bottles", "no more bottles"];
phrase(1)      -> ["1 bottle", "1 bottle", "no more bottles"];
phrase(2)      -> ["2 bottles", "2 bottles", "1 bottle"];
phrase(Bottle) -> lists:duplicate(2, integer_to_list(Bottle) ++ " bottles") ++ [integer_to_list(Bottle-1) ++ " bottles"].

bottles() -> lists:reverse(lists:seq(0,99)).

sing() ->
    lists:foreach(fun spawn_singer/1, bottles()),
    sing_verse(99).

spawn_singer(Bottle) ->
    Pid = self(), 
    spawn(fun() -> Pid ! create_verse(Bottle) end).

sing_verse(Bottle) ->
    receive
	{_, Verse} when Bottle == 0 ->
	    io:format(Verse);
	{N, Verse} when Bottle == N ->
	    io:format(Verse),
	    sing_verse(Bottle-1)
    after 
	3000 ->
	    io:format("Verse not received after 3 seconds - re-starting singer~n"),
	    spawn_singer(Bottle),
	    sing_verse(Bottle)
    end.

Here's a concurrent Elisp/Distel version:

(defun template (n)
  (if (= n 0)
      "%s of beer on the wall, %s of beer.nGo to the store and buy some more, 99 bottles of beer on the wall.n"
    "%s of beer on the wall, %s of beer.nTake one down and pass it around, %s of beer on the wall.nn"))

(defun create-verse (bottle)
  (let ((phrases (phrase bottle)))
    (list bottle (format (template bottle) (first phrases) (second phrases) (third phrases)))))

(defun phrase (bottle)
  (case bottle
    (0 (list "No more bottles" "no more bottles"))
    (1 (list "1 bottle" "1 bottle" "no more bottles"))
    (2 (list "2 bottles" "2 bottles" "1 bottle"))
    (t (list (format "%s bottles" bottle) (format "%s bottles" bottle) (format "%s bottles" (1- bottle))))))

(defun bottles ()
  (loop for i from 0 to 99 collecting i))

(defun sing ()
  (interactive)
  (erl-spawn
    (sing-verse 99)
    (dolist (bottle (bottles))
      (spawn-singer bottle))))

(defun spawn-singer (bottle)
  (erl-spawn
    (erl-send 'singer (create-verse bottle))))

(defun sing-verse (bottle)
  (erl-spawn
    (erl-register 'singer)
    (&sing-verse bottle nil)))

(defun &sing-verse (bottle song)
  (erl-receive (bottle song)
      ((verse (setq song (append (cdr verse) song))))
    (if (= bottle 0)
	(progn
	  (dotimes (b 100)
	    (if song (insert (pop song))))
	  (&sing-verse -1 nil))
      (&sing-verse (1- bottle) song))))

This code demonstrates how Distel has added Erlang-style concurrency features to Emacs Lisp. The spawn/send/receive commands are all being done in Elisp and there is no interaction with Erlang at all. It is easy to see the similarities between the Erlang code and the Elisp code and the way they deal with concurrency. Note that the Elisp/Distel version is spawning off separate "processes" (even though Emacs Lisp does not support processes/threads) to create the different verses. It also spawns off a sing-verse function that collects all of the verses and prints them out in the process buffer (which will be named "*reg singer*").


3. An interoperable Erlang/Elisp/Distel solution (with the Elisp/Distel node communicating with the Erlang node):


Here's a concurrent Erlang/Elisp/Distel version - first, the Erlang code (which only has the code necessary to create a verse of the song):

-module(beersongutil).
-export([create_verse/1]).

template(0)  -> "~s of beer on the wall, ~s of beer.~nGo to the store and buy some more, 99 bottles of beer on the wall.~n";
template(_N) -> "~s of beer on the wall, ~s of beer.~nTake one down and pass it around, ~s of beer on the wall.~n~n".

create_verse(0)      -> list_to_binary(io_lib:format(template(0), phrase(0)));
create_verse(Bottle) -> list_to_binary(io_lib:format(template(Bottle), phrase(Bottle))).

phrase(0)      -> ["No more bottles", "no more bottles"];
phrase(1)      -> ["1 bottle", "1 bottle", "no more bottles"];
phrase(2)      -> ["2 bottles", "2 bottles", "1 bottle"];
phrase(Bottle) -> lists:duplicate(2, integer_to_list(Bottle) ++ " bottles") ++ [integer_to_list(Bottle-1) ++ " bottles"].

then, the Elisp/Distel code:

(defun bottles ()
  (reverse (loop for i from 0 to 99 collecting i)))

(defun sing ()
  (interactive)
  (dolist (bottle (bottles))
    (erl-spawn
      (erl-send-rpc
       (erl-target-node)
       'beersongutil
       'create_verse
       (list bottle))
      (erl-receive ()
	  ((['rex verse] (pop-to-buffer "*scratch*")
	    (insert verse)))))))

To keep things simple (and to maintain consistency with the other examples), I haven't added in any code to automatically load the Erlang code into the Erlang node and to connect Distel to the Erlang node. If you're interested in running the example, you'll have to do something like the following on the Erlang side:

c(beersongutil). % compile the beersong Erlang utility functions in the node

And something like this in Emacs:

"M-x erl-choose-nodename" and specify the Erlang node that has the beersongutil code
load the above Elisp code
Run with "M-x sing"

The following screenshot shows the Elisp code, the Erlang code, the Erlang shell for the node that the Elisp code is communicating with, and the output: