{"id":602167,"date":"2023-01-28T07:48:59","date_gmt":"2023-01-28T13:48:59","guid":{"rendered":"https:\/\/news.sellorbuyhomefast.com\/index.php\/2023\/01\/28\/erlangs-not-about-lightweight-processes-and-message-passing\/"},"modified":"2023-01-28T07:48:59","modified_gmt":"2023-01-28T13:48:59","slug":"erlangs-not-about-lightweight-processes-and-message-passing","status":"publish","type":"post","link":"https:\/\/newsycanuse.com\/index.php\/2023\/01\/28\/erlangs-not-about-lightweight-processes-and-message-passing\/","title":{"rendered":"Erlang’s not about lightweight processes and message passing"},"content":{"rendered":"
\n
\n\n\n\n\n\n
date<\/th>\n<\/tr>\n<\/thead>\n
\n

2023-01-18<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Erlang’s not about lightweight processes and message passing…<\/h2>\n

I used to think that the big idea of Erlang is its lightweight processes and
\nmessage passing. Over the last couple of years I’ve realised that there’s a
\nbigger insight to be had, and in this post I’d like to share it with you.<\/p>\n

Background<\/h2>\n

Erlang has an interesting history. If I understand things correctly, it started
\noff as a Prolog library for building reliable distributed systems, morphed into a
\nProlog dialect, before finally becoming a language in its own right.<\/p>\n

The goal seemed to have always been to solve the problem of building reliable
\ndistributed systems. It was developed at Ericsson and used to program their
\ntelephone switches. This was sometime in the 80s and 90s, before internet use
\nbecome widespread. I suppose they were already dealing with “internet scale”
\ntraffic, i.e. hundreds of millions of users, with stricter SLAs than most
\ninternet services provide today. So in a sense they were ahead of their time.<\/p>\n

In 1998 Ericsson decided to ban all use of Erlang1<\/a><\/sup>. The people responsible
\nfor developing it argued that if they were going to ban it, then they might as
\nwell open source it. Which Ericsson did and shortly after most of the team that
\ncreated Erlang quit and started their own company.<\/p>\n

One of these people was Joe Armstrong, which also was one of the main people
\nbehind the design and implementation of Erlang. The company was called Bluetail
\nand they got bought up a couple of times but in the end Joe got fired in 2002.<\/p>\n

Shortly after, still in 2002, Joe starts writing his PhD thesis at the Swedish
\nInstitute of Computer Science (SICS). Joe was born 1950, so he was probably 52
\nyears old at this point. The topic of the thesis is Making reliable distributed
\nsystems in the presence of software errors<\/em> and it was finished the year after
\nin 2003.<\/p>\n

It’s quite an unusual thesis in many ways. For starters, most theses are written
\nby people in their twenties with zero experience of practical applications.
\nWhereas in Joe’s case he has been working professionally on this topic since the
\n80s, i.e. about twenty years. The thesis contains no math nor theory, it’s
\nmerely a presentation of the ideas that underpin Erlang and how they used Erlang
\nto achieve the original goal of building reliable distributed systems.<\/p>\n

I highly commend reading his
\nthesis<\/a>
\nand forming your own opinion, but to me it’s clear that the big idea there isn’t
\nlightweight processes2<\/a><\/sup> and message passing, but rather the generic components
\nwhich in Erlang are called behaviours<\/em>.<\/p>\n

Behaviours<\/h2>\n

I’ll first explain in more detail what behaviours are, and then I’ll come back
\nto the point that they are more important than the idea of lightweight processes.<\/p>\n

Erlang behaviours are like interfaces in, say, Java or Go. It’s a collection of
\ntype signatures which can have multiple implementations, and once the programmer
\nprovides such an implementation they get access to functions written against
\nthat interface. To make it more concrete here’s a contrived example in Go:<\/p>\n

<\/p>\n
\/\/ The interface.<\/span>\ntype<\/span> HasName<\/span> interface<\/span> {\n        Name<\/span>() string<\/span>\n}\n\n\/\/ A generic function written against the interface.<\/span>\nfunc<\/span> Greet<\/span>(n<\/span> HasName<\/span>) {\n    fmt<\/span>.Printf<\/span>(\"Hello %s!n<\/span>\"<\/span>, n<\/span>.Name<\/span>())\n}\n\n\/\/ First implementation of the interface.<\/span>\ntype<\/span> Joe<\/span> struct<\/span> {}\n\nfunc<\/span> (_<\/span> *<\/span>Joe<\/span>) Name<\/span>() string<\/span> {\n        return<\/span> \"Joe\"<\/span>\n}\n\n\/\/ Second implementation of the interface.<\/span>\ntype<\/span> Mike<\/span> struct<\/span> {}\n\nfunc<\/span> (_<\/span> *<\/span>Mike<\/span>) Name<\/span>() string<\/span> {\n        return<\/span> \"Mike\"<\/span>\n}\n\nfunc<\/span> main<\/span>() {\n        joe<\/span> :=<\/span> &<\/span>Joe<\/span>{}\n        mike<\/span> :=<\/span> &<\/span>Mike<\/span>{}\n        Greet<\/span>(mike<\/span>)\n        Greet<\/span>(joe<\/span>)\n}<\/pre>\n<\/div>\n
Running the above program will display:<\/p>\n
This hopefully illustrates how Greet<\/code> is generic in, or parametrised by, the
\ninterface HasName<\/code>.<\/p>\n
Generic server behaviour<\/h3>\n
Next lets have look at a more complicated example in Erlang taken from Joe’s
\nthesis (p. 136). It’s a key-value store where we can store<\/code> a key value pair or
\nlookup<\/code> the value of a key, the handle_call<\/code> part is the most interesting:<\/p>\n

\n  gen_server:start_link({local,kv},kv,arg1,[]).<\/p>\n
stop() -> gen_server:cast(kv, stop).<\/p>\n
init(arg1) ->
\n  io:format(“Key-Value server starting~n”),
\n  {ok, dict:new()}.<\/p>\n
store(Key, Val) ->
\n  gen_server:call(kv, {store, Key, Val}).<\/p>\n
lookup(Key) -> gen_server:call(kv, {lookup, Key}).<\/p>\n
handle_call({store, Key, Val}, From, Dict) ->
\n  Dict1 = dict:store(Key, Val, Dict),
\n  {reply, ack, Dict1};
\nhandle_call({lookup, crash}, From, Dict) ->
\n  1\/0; %% <- deliberate error :-)\nhandle_call({lookup, Key}, From, Dict) ->
\n  {reply, dict:find(Key, Dict), Dict}.<\/p>\n
handle_cast(stop, Dict) -> {stop, normal, Dict}.<\/p>\n
terminate(Reason, Dict) ->
\n  io:format(“K-V server terminating~n”).”><\/p>\n
-module<\/span>(kv<\/span>).\n-behaviour<\/span>(gen_server<\/span>).\n\n-export<\/span>([start<\/span>\/0<\/span>, stop<\/span>\/0<\/span>, lookup<\/span>\/1<\/span>, store<\/span>\/2<\/span>]).\n\n-export<\/span>([init<\/span>\/1<\/span>, handle_call<\/span>\/3<\/span>, handle_cast<\/span>\/2<\/span>, terminate<\/span>\/2<\/span>]).\n\nstart<\/span>() -><\/span>\n  gen_server<\/span>:start_link<\/span>({local<\/span>,kv<\/span>},kv<\/span>,arg1<\/span>,[]).\n\nstop<\/span>() -><\/span> gen_server<\/span>:cast<\/span>(kv<\/span>, stop<\/span>).\n\ninit<\/span>(arg1<\/span>) -><\/span>\n  io<\/span>:format<\/span>(\"<\/span>Key-Value server starting~n<\/span>\"<\/span><\/span>),\n  {ok<\/span>, dict<\/span>:new<\/span>()}.\n\nstore<\/span>(Key<\/span>, Val<\/span>) -><\/span>\n  gen_server<\/span>:call<\/span>(kv<\/span>, {store<\/span>, Key<\/span>, Val<\/span>}).\n\nlookup<\/span>(Key<\/span>) -><\/span> gen_server<\/span>:call<\/span>(kv<\/span>, {lookup<\/span>, Key<\/span>}).\n\nhandle_call<\/span>({store<\/span>, Key<\/span>, Val<\/span>}, From<\/span>, Dict<\/span>) -><\/span>\n  Dict1<\/span> =<\/span> dict<\/span>:store<\/span>(Key<\/span>, Val<\/span>, Dict<\/span>),\n  {reply<\/span>, ack<\/span>, Dict1<\/span>};\nhandle_call<\/span>({lookup<\/span>, crash<\/span>}, From<\/span>, Dict<\/span>) -><\/span>\n  1<\/span>\/<\/span>0<\/span>; %<\/span>% <- deliberate error :-)<\/span>\nhandle_call<\/span>({lookup<\/span>, Key<\/span>}, From<\/span>, Dict<\/span>) -><\/span>\n  {reply<\/span>, dict<\/span>:find<\/span>(Key<\/span>, Dict<\/span>), Dict<\/span>}.\n\nhandle_cast<\/span>(stop<\/span>, Dict<\/span>) -><\/span> {stop<\/span>, normal<\/span>, Dict<\/span>}.\n\nterminate<\/span>(Reason<\/span>, Dict<\/span>) -><\/span>\n  io<\/span>:format<\/span>(\"<\/span>K-V server terminating~n<\/span>\"<\/span><\/span>).<\/pre>\n<\/div>\n
This is an implementation of the gen_server<\/code> behaviour\/interface. Notice how
\nhandle_call<\/code> updates the state (Dict<\/code>) in case of a store<\/code> and lookup<\/code>s the
\nkey in the state. Once gen_server<\/code> is given this implementation it will provide
\na server which can handle concurrent store<\/code> and lookup<\/code> requests, similarly to
\nhow Greet<\/code> provided the displaying functionality.<\/p>\n
At this point you might be thinking “OK, so what? Lots of programming languages
\nhave interfaces…”. That’s true, but notice how handle_call<\/code> is completely
\nsequential, i.e. all concurrency is hidden away in the generic gen_server<\/code>
\ncomponent. “Yeah, but that’s just good engineering practice which can be done in
\nany language” you say. That’s true as well, but the thesis pushes this idea
\nquite far. It identifies six behaviours: gen_server<\/code>, gen_event<\/code>, gen_fsm<\/code>,
\nsupervisor<\/code>, application<\/code>, and release<\/code> and then says these are enough to
\nbuild reliable distributed systems. As a case study Joe uses one of Ericsson’s
\ntelephone switches (p. 157):<\/p>\n
\n
When we look at the AXD301 project in chapter 8, we will see that there were
\n122 instances of gen_server, 36 instances of gen_event and 10 instances of
\ngen_fsm. There were 20 supervisors and 6 applications. All this is packaged
\ninto one release.<\/p>\n<\/blockquote>\n
Joe gives several arguments for why behaviour should be used (p. 157-158):<\/p>\n
    \n
  1. \n
    The application programmer only has to provide the part of the code which
    \ndefines the semantics<\/em> (or “business logic”) of their problem, while the
    \ninfrastructure<\/em> code is provided automatically by the behaviour;<\/p>\n<\/li>\n
  2. \n
    The application programmer writes sequential code, all concurrency is hidden
    \naway in the behaviour;<\/p>\n<\/li>\n
  3. \n
    Behaviours are written by experts, and based on years of experience and
    \nrepresent “best practices”;<\/p>\n<\/li>\n
  4. \n
    Easier for new team members to get started: business logic is sequential,
    \nsimilar structure that they might have seen before elsewhere;<\/p>\n<\/li>\n
  5. \n
    If whole systems are implemented reusing a small set of behaviours: as
    \nbehaviour implementations improve the whole systems will improve without
    \nrequiring any code changes;<\/p>\n<\/li>\n
  6. \n
    Sticking to only using behaviours enforces structure, which in turn makes
    \ntesting and formal verification much easier.<\/p>\n<\/li>\n<\/ol>\n
    We’ll come back to this last point about testing later.<\/p>\n
    Event manager behaviour<\/h3>\n
    Lets come back to the behaviours we listed above first. We looked at
    \ngen_server<\/code>, but what are the others for? There’s gen_event<\/code> which is a
    \ngeneric event manager, which lets you register event handlers that are then run
    \nwhen the event manager gets messages associated with the handlers. Joe says this
    \nis useful for, e.g., error logging and gives the following example of an simple
    \nlogger (p. 142):<\/p>\n

    \n  case gen_event:start_link({local, ?NAME}) of
    \n    Ret = {ok, Pid} ->
    \n      gen_event:add_handler(?NAME,?MODULE,arg1),
    \n      Ret;
    \n  Other ->
    \n    Other
    \n  end.<\/p>\n
    stop() -> gen_event:stop(?NAME).<\/p>\n
    log(E) -> gen_event:notify(?NAME, {log, E}).<\/p>\n
    report() ->
    \n  gen_event:call(?NAME, ?MODULE, report).<\/p>\n
    init(arg1) ->
    \n  io:format(“Logger starting~n”),
    \n  {ok, []}.<\/p>\n
    handle_event({log, E}, S) -> {ok, trim([E|S])}.<\/p>\n
    handle_call(report, S) -> {ok, S, S}.<\/p>\n
    terminate(stop, _) -> true.<\/p>\n
    trim([X1,X2,X3,X4,X5|_]) -> [X1,X2,X3,X4,X5];
    \ntrim(L) -> L.”><\/p>\n
    -module<\/span>(simple_logger<\/span>).\n-behaviour<\/span>(gen_event<\/span>).\n\n-export<\/span>([start<\/span>\/0<\/span>, stop<\/span>\/0<\/span>, log<\/span>\/1<\/span>, report<\/span>\/0<\/span>]).\n\n-export<\/span>([init<\/span>\/1<\/span>, terminate<\/span>\/2<\/span>,\n         handle_event<\/span>\/2<\/span>, handle_call<\/span>\/2<\/span>]).\n\n-define<\/span>(NAME<\/span>, my_simple_event_logger<\/span>).\n\nstart<\/span>() -><\/span>\n  case<\/span> gen_event<\/span>:start_link<\/span>({local<\/span>, ?<\/span>NAME<\/span>}) of<\/span>\n    Ret<\/span> =<\/span> {ok<\/span>, Pid<\/span>} ->\n      gen_event<\/span>:add_handler<\/span>(?<\/span>NAME<\/span>,?<\/span>MODULE<\/span>,arg1<\/span>),\n      Ret<\/span>;\n  Other<\/span> ->\n    Other<\/span>\n  end<\/span>.\n\nstop<\/span>() -><\/span> gen_event<\/span>:stop<\/span>(?<\/span>NAME<\/span>).\n\nlog<\/span>(E<\/span>) -><\/span> gen_event<\/span>:notify<\/span>(?<\/span>NAME<\/span>, {log<\/span>, E<\/span>}).\n\nreport<\/span>() -><\/span>\n  gen_event<\/span>:call<\/span>(?<\/span>NAME<\/span>, ?<\/span>MODULE<\/span>, report<\/span>).\n\ninit<\/span>(arg1<\/span>) -><\/span>\n  io<\/span>:format<\/span>(\"<\/span>Logger starting~n<\/span>\"<\/span><\/span>),\n  {ok<\/span>, []}.\n\nhandle_event<\/span>({log<\/span>, E<\/span>}, S<\/span>) -><\/span> {ok<\/span>, trim<\/span>([E<\/span>|S<\/span>])}.\n\nhandle_call<\/span>(report<\/span>, S<\/span>) -><\/span> {ok<\/span>, S<\/span>, S<\/span>}.\n\nterminate<\/span>(stop<\/span>, _<\/span>) -><\/span> true<\/span>.\n\ntrim<\/span>([X1<\/span>,X2<\/span>,X3<\/span>,X4<\/span>,X5<\/span>|_<\/span>]) -><\/span> [X1<\/span>,X2<\/span>,X3<\/span>,X4<\/span>,X5<\/span>];\ntrim<\/span>(L<\/span>) -><\/span> L<\/span>.<\/pre>\n<\/div>\n
    The interesting part is handle_event<\/code>, trim<\/code> and report<\/code>. Together they let
    \nthe user log, keep track and display the last five error messages.<\/p>\n
    State machine behaviour<\/h3>\n
    The gen_fsm<\/code> behavior has been renamed to gen_statem<\/code> (for state machine)
    \nsince thesis was written. It’s very similar to gen_server<\/code>, but more geared
    \ntowards implementing protocols, which often are specified as state machines. I
    \nbelieve any gen_server<\/code> can be implemented as a gen_statem<\/code> and vice versa so
    \nwe won’t go into the details of gen_statem<\/code>.<\/p>\n
    Supervisor behaviour<\/h3>\n
    The next interesting behavior is supervisor<\/code>. Supervisors are processes which
    \nsole job is to make sure that other processes are healthy and doing their job.
    \nIf a supervised process fails then the supervisor can restart it according
    \nto some predefined strategy. Here’s an example due to Joe (p. 148):<\/p>\n

    \n  supervisor:start_link({local, simple_supervisor},
    \n  ?MODULE, nil).<\/p>\n
    init(_) ->
    \n  {ok,
    \n  {{one_for_one, 5, 1000},
    \n  [
    \n   {packet,
    \n     {packet_assembler, start, []},
    \n     permanent, 500, worker, [packet_assembler]},
    \n   {server,
    \n     {kv, start, []},
    \n     permanent, 500, worker, [kv]},
    \n   {logger,
    \n     {simple_logger, start, []},
    \n     permanent, 500, worker, [simple_logger]}]}}.”><\/p>\n
    -module<\/span>(simple_sup<\/span>).\n-behaviour<\/span>(supervisor<\/span>).\n\n-export<\/span>([start<\/span>\/0<\/span>, init<\/span>\/1<\/span>]).\n\nstart<\/span>() -><\/span>\n  supervisor<\/span>:start_link<\/span>({local<\/span>, simple_supervisor<\/span>},\n  ?<\/span>MODULE<\/span>, nil<\/span>).\n\ninit<\/span>(_<\/span>) -><\/span>\n  {ok<\/span>,\n  {{one_for_one<\/span>, 5<\/span>, 1000<\/span>},\n  [\n   {packet<\/span>,\n     {packet_assembler<\/span>, start<\/span>, []},\n     permanent<\/span>, 500<\/span>, worker<\/span>, [packet_assembler<\/span>]},\n   {server<\/span>,\n     {kv<\/span>, start<\/span>, []},\n     permanent<\/span>, 500<\/span>, worker<\/span>, [kv<\/span>]},\n   {logger<\/span>,\n     {simple_logger<\/span>, start<\/span>, []},\n     permanent<\/span>, 500<\/span>, worker<\/span>, [simple_logger<\/span>]}]}}.<\/pre>\n<\/div>\n
    The {one_for_one, 5, 1000}<\/code> is the restart strategy. It says that if one of the
    \nsupervised processes (packet_assembler<\/code>, kv<\/code>, and simple_logger<\/code>) fail then
    \nonly restart the failing process (one_for_one<\/code>). If the supervisor needs to
    \nrestart more than 5<\/code> times in 1000<\/code> seconds then the supervisor itself should
    \nfail.<\/p>\n
    The permanent, 500, worker<\/code> part means that this is a worker process which
    \nshould be permanently kept alive and its given 500 milliseconds to gracefully
    \nstop what it’s doing in case the supervisor wants to restart it.<\/p>\n
    “Why would the supervisor want to restart it if it’s not dead already?”, one
    \nmight wonder. Well, there are other restart strategies than one_for_one<\/code>. For
    \nexample, one_for_all<\/code> where if one process fails then the supervisor restarts
    \nall of its children.<\/p>\n
    If we also consider that supervisors can supervise supervisors, which are not
    \nnecessarily running on the same computer, then I hope that you get an idea of
    \npowerful this behaviour can be. And, no, this isn’t “just Kubernetes”, because
    \nit’s at the thread\/lightweight process level rather than docker container level.<\/p>\n
    The idea for supervisors and their restart strategies comes from the observation
    \nthat often a restart appears to fix the problem, as captured in the Have You
    \nTried Turning It Off And On Again?<\/em> sketches from IT Crowd.<\/p>\n
    Knowing that failing processes will get restarted coupled with Jim Gray’s idea
    \nof failing fast, that’s either produce the output according to the specification
    \nor signal failure and stop operating, leads to Joe’s slogan: “Let it crash!” (p.
    \n107). Another way to think of it is that a program should only express its
    \n“happy path”, should anything go wrong on its happy way it should crash, rather
    \nthan trying to be clever about it and try to fix the problem (potentially making
    \nit worse), and another program higher up the supervisor tree will handle it.<\/p>\n
    Supervisors and the “let it crash” philosophy, appear to produce reliable
    \nsystems. Joe uses the Ericsson AXD301 telephone switch example again (p. 191):<\/p>\n
    \n
    Evidence for the long-term operational stability of the system had also not
    \nbeen collected in any systematic way. For the Ericsson AXD301 the only
    \ninformation on the long-term stability of the system came from a power-point
    \npresentation showing some figures claiming that a major customer had run an 11
    \nnode system with a 99.9999999% reliability, though how these figure had been
    \nobtained was not documented.<\/p>\n<\/blockquote>\n
    To put this in perspective, five nines (99.999%) reliability is considered good
    \n(5.26 minutes of downtime per year). “59% of Fortune 500 companies experience a
    \nminimum of 1.6 hours of downtime per week”, according to some
    \nreport<\/a>
    \nfrom a biased company. Notice per year<\/em> vs per week<\/em>, but as we don’t know how
    \neither reliability numbers are obtained its probably safe to assume that the
    \ntruth is somewhere in the middle — still a big difference, but not 31.56
    \nmilliseconds (nine nines) of downtime per year vs 1.6 hours of downtime per
    \nweek.<\/p>\n
    Application and release behaviours<\/h3>\n
    I’m not sure if application<\/code> and release<\/code> technically are behaviours, i.e.
    \ninterfaces. They are part of the same chapter as the other behaviours in the
    \nthesis and they do provide a clear structure which is a trait of the other
    \nbehaviours though, so we’ll include them in the discussion.<\/p>\n
    So far we’ve presented behaviours from the bottom up. We started with “worker”
    \nbehaviours gen_server<\/code>, gen_statem<\/code> and gen_event<\/code> which together capture the
    \nsemantics of our problem. We then saw how we can define supervisor<\/code> trees whose
    \nchildren are other supervisor trees or workers, to deal with failures and
    \nrestarts.<\/p>\n
    Next level up is an application<\/code> which consists of a supervisor tree together
    \nwith everything else we need to deliver a particular application.<\/p>\n
    A system can consist of several application<\/code> and that’s where the final
    \n“behaviour” comes in. A release<\/code> packages up one or more applications. They
    \nalso contain code to handle upgrades. If the upgrade fails, it must be able to
    \nrollback to the previous stable state.<\/p>\n
    How behaviours can be implemented<\/h2>\n
    I hope that by now I’m managed to convince you that it’s not actually the
    \nlightweight processes and message passing by themselves that make Erlang great
    \nfor building reliable systems.<\/p>\n
    At best one might be able to claim that lightweight processes and supervisors
    \nare the key mechanisms at play    3<\/a><\/sup>, but I think it would be more honest to recognise
    \nthe structure that behaviours provide and how that ultimately leads to reliable
    \nsoftware.<\/p>\n
    I’ve not come across any other language, library, or framework which provides
    \nsuch relatively simple building blocks that compose into big systems like the
    \nAXD301 (“over a million lines of Erlang code”, p. 167).<\/p>\n
    This begs the question: why aren’t language and library designers stealing the
    \nstructure behind Erlang’s behaviours, rather than copying the ideas of
    \nlightweight processes and message passing?<\/p>\n
    Let’s take a step back. We said earlier that behaviours are interfaces and many
    \nprogramming languages have interfaces. How would we go about starting to
    \nimplement behaviours in other languages?<\/p>\n
    Lets start with gen_server<\/code>. I like to think its interface signature as being:<\/p>\n
     State -> (State, Output)”><\/p>\n
    Input<\/span> -><\/span> State<\/span> -><\/span> (State<\/span>, Output<\/span>)<\/pre>\n<\/div>\n
    That’s it takes some input, its current state and produces a pair of the new
    \nupdated state and an output.<\/p>\n
    How do we turn this sequential signature into something that can handle
    \nconcurrent requests? One way would be to fire up a HTTP server which transforms
    \nrequests into Input<\/code>s and puts them on a queue, have an event loop which pops
    \ninputs from the queue and feeds it to the sequential implementation, then
    \nwriting the output back to the client response. It wouldn’t be difficult to
    \ngeneralise this to be able to handle multiple gen_server<\/code>s at the same time, by
    \ngiving each a name and let the request include the name in addition to the
    \ninput.<\/p>\n
    gen_event<\/code> could be implemented by allowing registration of callbacks to
    \ncertain types of event on the queue.<\/p>\n
    supervisor<\/code>s is more interesting, one simple way to think of it is: when we
    \nfeed the gen_server<\/code> function the next input from the queue, we wrap that call
    \nin an exception handler, and should it throw we notify its supervisor. It gets a
    \nbit more complicated if the supervisor is not running on the same computer as
    \nthe gen_server<\/code>.<\/p>\n
    I haven’t thought about application<\/code> and release<\/code>s much yet, but given that
    \nconfiguration, deployment and upgrades are difficult problems they seem
    \nimportant.<\/p>\n
    Correctness of behaviours<\/h2>\n
    Writing a post solely about stealing from Erlang doesn’t seem fair, even though
    \nit’s the right thing to do, so I’d like to finish off with how we can build upon
    \nthe insights of Joe and the Erlang community.<\/p>\n
    I’ve been interesting in testing for a while now. Most recently I’ve been
    \nlooking into     simulation
    \ntesting<\/a>
    \ndistributed systems \u00e0 la
    \n    FoundationDB<\/a>.<\/p>\n
    Simulation testing in a nutshell is running your system in a simulated world,
    \nwhere the simulation has full control over which messages get sent when over the
    \nnetwork.<\/p>\n
    FoundationDB built their own programming language, or dialect of C++ with
    \nactors, in order do the simulation testing. Our team seemed to be able to get
    \nquite far with merely using state machines of type:<\/p>\n
     State -> (State, [Output])”><\/p>\n
    Input<\/span> -><\/span> State<\/span> -><\/span> (State<\/span>, [Output<\/span>])<\/pre>\n<\/div>\n
    where [Output]<\/code> is a sequence of outputs.<\/p>\n
    The idea being that the simulator keeps track of a priority queue of messages
    \nsorted by their arrival time, it pops a message, advances the clock to the
    \narrival time of that message, feeds the message to the receiving state machine,
    \ngenerates new arrival times for all output messages and puts them back into the
    \npriority queue, rinse and repeat. As long as everything is deterministic and the
    \narrival times are generated using a seed we can explore many different
    \ninterleavings and get reproducible failures. It’s also much faster than Jepsen,
    \nbecause messaging is done in-memory and we advance the clock to the arrival
    \ntime, thereby triggering any timeouts without having to wait for them.<\/p>\n
    We used to say that programs of this state machine type where written in
    \n“network normal form”, and conjectured that every program which can receive and
    \nsend stuff over the network can be refactored into this shape    4<\/a><\/sup>. Even if we
    \nhad a proof, “network normal form” always felt a bit arbitrary. But then I read
    \nJoe’s thesis and realised that gen_server<\/code> and gen_statem<\/code> basically have the
    \nsame type, so I stopped being concerned about it. As I think that if a structure
    \nis found to be useful by different people, then it’s usually a sign that it
    \nisn’t arbitrary.<\/p>\n
    Anyway, in, at least, one of Joe’s talks<\/a>
    \nhe mentions how difficult it’s to correctly implement distributed leader
    \nelection.<\/p>\n
    I believe this is a problem that would be greatly simplified by having access to
    \na simulator. A bit like I’d imagine having access to a wind tunnel would make
    \nbuilding an airplane easier. Both lets you test your system under extreme
    \nconditions, such as unreliable networking or power loss, before they happen in
    \n“production”. Furthermore, this simulator can be generic in, or parametrised by,
    \nbehaviours. Which means that the developer gets it for free while the complexity
    \nof the simulator is hidden away, just like the concurrent code of gen_server<\/code>!<\/p>\n
    FoundationDB is a good example of simulation testing working, as witnessed by
    \nthis     tweet<\/a> where somebody
    \nasked Kyle “aphyr” Kingsbury to Jepsen test FoundationDB:<\/p>\n
    \n
    \u201chaven\u2019t tested foundation[db] in part because their testing appears to be
    \nwaaaay more rigorous than mine.\u201d<\/p>\n<\/blockquote>\n
    Formal verification is also made easier if the program is written a state
    \nmachine. Basically all of Lamport’s model checking
    \n    work<\/a>
    \nwith TLA+ assumes that the specification is a state machine. Also more recently
    \nKleppmann has
    \n    shown<\/a>
    \nhow to exploit the state machine structure to do proof by (structural) induction
    \nto solve the state explosion problem.<\/p>\n
    So there you have it, we’ve gone full circle. We started by taking inspiration
    \nfrom Joe and Erlang’s behaviours, and ended up using the structure of the
    \ngen_server<\/code> behaviour to make it easier to solve a problem that Joe used to
    \nhave.<\/p>\n
    Contributing<\/h2>\n
    There are a bunch of related ideas that I have started working on:<\/p>\n