Dan Swain

about me

Backend developer at simpli.fi. Robotics Ph.D. I like building things.

 

found on

Generating an Erlang sys.config From an Elixir config.exs

- - posted in chef, deployment, elixir, erlang

This post describes Erlang’s sys.config and the equivalent config.exs in Elixir and how you can convert from config.exs to sys.config (the link is to a script that does this). I also talk a little about how you could use this to deploy Elixir projects using Chef.

Erlang’s sys.config

Erlang releases use a file called sys.config for configuration values. This is the place to put your OTP applications’ configuration in addition to configuration for system applications like the logger and sasl. You can read these values in your code using application:get_env/2. A sys.config file is just an Erlang term; application:get_env assumes that it is a proplist.

As an example, consider login credentials for an outside service. We could store these in the sys.config file (rel/files/sys.config if we’re building using rebar) as something like the following.

1
2
3
4
5
6
7
8
% rel/files/sys.config
[
  {outside_service, [
      {host, <<"dantswain.com">>},
      {user_name, <<"dantswain">>},
      {password, <<"s3kr4t">>}
    ]}
].

We would then access this config in our application using the following.

1
2
3
4
5
get_credentials() ->
  Host     = application:get_env(outside_service, host),
  UserName = application:get_env(outside_service, user_name),
  Password = application:get_env(outside_service, password),
  {Host, UserName, Password}.

Note that this approach only works for one environment. To use different credentials for testing, development, and production, we’d need to either come up with a config layout that allowed for multiple environments or swap out different files depending on the build environment. Either way is a bit cumbersome.

Elixir’s config.exs

Elixir enriches our ability to configure OTP applications via the Mix.Config DSL. This makes it very easy to have per-environment configurations (e.g., testing, development, and production) and to compose config files programmatically.

Since we’re going to talk specifically about deployment to a production environment, let’s make our config/config.exs read from different files depending on Mix.env, which we can set with the MIX_ENV environment variable.

1
2
3
4
# config/config.exs
use Mix.Config

import_config "#{Mix.env}.exs"

That is, we just defer to the per-environment config file. Our development config might point to localhost.

1
2
3
4
5
6
7
# config/dev.exs
use Mix.Config

config :outside_service,
  host: "localhost",
  user_name: "dantswain",
  password: "password"

Our testing config might use different credentials.

1
2
3
4
5
6
7
# config/test.exs
use Mix.Config

config :outside_service,
  host: "localhost",
  user_name: "test",
  password: "test"

We have to at least have a config/prod.exs file to be able to compile the project, but we’ll end up using a different method below to populate the production config at deployment time. So we can have a dummy config file in source control.

1
2
3
4
5
6
# config/prod.exs
use Mix.Config

# We generate production config during deployment, but
#  this file needs to exist so that mix does not fail when
#  it tries to import_config("prod.exs")

Configuration values are accessed using Application.get_env/3.

1
2
3
4
5
6
def get_credentials do
  host = Application.get_env(:outside_service, :host)
  user_name = Application.get_env(:outside_service, :user_name)
  password = Application.get_env(:outside_service, :password)
  {host, user_name, password}
end

Configuration management

I highly recommend exrm to build releases for deploying Elixir projects. I highly recommend using a configuration management tool for deployment in general. I’m a fan of Chef, but most of this post isn’t specific to Chef.

exrm adds a Mix task called mix release to your project. To build a production release, you just run MIX_ENV=prod mix release (after the usual build cycle). This step creates an Erlang release, which contains everything needed to run your app. Inside the release is a sys.config file (releases/<version>/sys.config). Application.get_env/3 will consult this file when your code is deployed.

Using mix release out-of-the-box will convert your config/config.exs into a sys.config file. This is fine if you are not using configuration management, but you should really be using configuration management ;). exrm handles this using a tool called conform, which allows you to place .conf files inside your release. conform automatically builds a new sys.config file each time you start your app (bin/my_app start).

conform is probably The Right Way To Do It. I’m going to present a simpler method here that also works. This might be helpful to you if you don’t have time to grok conform, or it might just be useful to you as a way of understanding part of what exrm and conform do under the hood.

If you look at the exrm source code, you’ll see that exrm uses Mix.Config.read!/1 to read in the appropriate config file and then simply formats and writes the config term to a file. The implementation of Mix.Config.read!/1 is itself fairly straightforward and relies on Code.eval_file/2 to turn the .exs files into terms.

Converting a config.exs to a sys.config is essentially a matter of using Mix.Config.read!/1 to obtain a term and then writing it to disk. It could be implemented simply using something like this.

1
2
3
config = Mix.Config.read!("config.exs")
sys_config_string = :io_lib.format('~p.~n', [config]) |> List.to_string
File.write("sys.config", sys_config_string)

I wrote a more fleshed-out Elixir script to generate a sys.config file from a config.exs file that includes some error handling and usage messages.

Deploying an Elixir project using Chef

This is how I use Chef to deploy Elixir applications. I won’t go go into detail on how to install Erlang, Elixir, Mix, etc. I also won’t cover how to build your code, which is fairly straightforward (the usual mix deps.get, mix compile, etc., but with MIX_ENV=prod).

  1. Build your release using exrm’s MIX_ENV=prod mix release. Use a build server if you have one, otherwise build in a temporary location. The output of the build process is a .tar.gz file of your release.
  2. Untar the release file into the deployment location and set appropriate ownership and permissions.
  3. Use a Chef template to generate a config.exs file for your particular deployment environment. Have Chef install the config.exs file alongside the release in releases/<version>/config.exs.
  4. Have Chef install the conversion script in the deployment’s PATH and make sure that it is executable.
  5. In the directory where the config.exs file lives (which is where you want the sys.config file to end up), execute MIX_ENV=prod elixir_to_sys_config config.exs > sys.config.

This way you can template your config.exs file in a way that is familiar to Elixir developers, and end up with a sys.config file that will work with your release app. If you structure your Chef cookbooks appropriately, you should be able to regenerate the config files without rebuilding the entire application.

Integration Testing a JSON API in Phoenix

- - posted in elixir, phoenix

TL;DR

To do integration testing in Phoenix, first enable the app’s content serving in its config, then use an HTTP client library (I use HTTPoison below) to make requests. The source code corresponding to this post is on my github at https://github.com/dantswain/calculon.

Introduction (plus a little discourse on testing)

I like integration-level testing, especially for web apps and doubly so for APIs. A good integration test suite can help you work out usability issues with your API, give you confidence that your deployment is stable, and provide a springboard for troubleshooting issues down the road. Publishing your test suite can also be a decent way to document it. If you’re working in Ruby, there is actually a gem to document your API through rspec (which I highly recommend). I’m not currently aware of an equivalent package for Elixir, though that would be awesome.

I was recently working on an API for a personal project using the Phoenix framework for Elixir and realized there wasn’t much guidance (at least as of the time of writing) on writing integration tests. I was able to figure it out after a little googling, so I thought I’d write up what I’ve learned.

This post covers integration testing of a JSON API. It’s theoretically quite possible to extend the same ideas to test a generic web app, though it could be tricky to handle Javascript.

Note: Phoenix hasn’t yet reached a 1.0 release version and is correspondingly in a little bit of flux between minor versions. This post was written using Phoenix v0.11.0 running on Elixir v1.0.2. I know there are a couple differences from v0.10.0 to v0.11.0 that will probably break this; see the note at the bottom of this post for at least one difference that I know of.

It’s just curling (kind of)

In theory, you could write integration tests for your API using any language and framework you want. You could script the tests in the shell using curl, or write a test suite in Ruby with rspec. All we need to do is make HTTP requests and validate the responses.

However, we’re already using Elixir and Elixir provides a pretty good testing framework called ExUnit. Furthermore, there’s probably code in our application that will help us a lot when writing tests - model code is useful for generating fixture data, etc. We may also want to validate the state of the application after a request.

Aside: In an ideal world, integration tests should (debatably) share nothing with your application code. That’s because you want integration tests not to depend on your implementation details. Using model code to set up fixtures and validate application state is a fairly reasonable compromise because it generally makes us a lot more productive.

Setting up an example app

Let’s build an app that does something very simple so that we can focus on the mechanics of integration testing rather than the implementation details. We aren’t doing any database calls since that would add a fair amount of complexity to the code and wouldn’t help us understand integration testing a Phoenix app. Nothing that we do here will preclude us from making and testing database calls later on, so let’s just avoid it.

We’ll make an app that performs simple arithmetic calculations. The goal is to be able to POST a calculation to the server and have the response include the result of the calculation. We’ll call the app Calculon.

Follow the usual process to set up a Phoenix app. As of v0.11.0, you have to run mix phoenix.new from within the installer directory of the Phoenix repo (as of the time of writing, the documentation hasn’t been updated).

1
2
in ~/src/phoenix/installer (v0.11.0)
$ mix phoenix.new ../../calculon

Set up an endpoint so that we can POST /api/v1/calculations:

1
2
3
4
5
6
# inside web/router.ex
scope "/api/v1", Calculon do
  pipe_through :api

  post "/calculations", CalculationsController, :create
end

Then add the corresponding controller:

1
2
3
4
5
6
7
8
9
10
11
12
# web/controllers/calculations_controller.ex
defmodule Calculon.CalculationsController do
  use Phoenix.Controller

  plug :action

  def create(conn, _params) do
    conn
    |> put_status(201)
    |> json  %{ok: true}
  end
end

This controller doesn’t really do anything at this point, it just handles the request and sets the HTTP status code to 201. We’ll add functionality later.

We can now run our app with mix phoenix.server and use curl to verify that it’s working:

1
2
3
4
5
6
7
8
9
10
$ curl -v -XPOST -d'{"calculation": { "input": "1 + 1" }}' 'http://127.0.0.1:4000/api/v1/calculations'
# (trimmed output)
# ...
* Connected to 127.0.0.1 (127.0.0.1) port 4000 (#0)
> POST /api/v1/calculations HTTP/1.1
# ...
* upload completely sent off: 37 out of 37 bytes
< HTTP/1.1 201 Created
# ...
{"ok":true}%

We get back an HTTP 201 with {"ok": true} as the body, which is exactly what we should get.

HTTPoison for requests

HTTPoison is an Elixir library for performing HTTP requests. Add it to the deps function in your mix.exs.

1
2
3
4
5
# inside mix.exs
defp deps do
  [ #... other deps
  {:httpoison, "~> 0.6"}]
end

Then run mix deps.get and mix deps.compile.

Now let’s add a helper module to make API calls, making heavy use of HTTPoison’s HTTPoison.Base macro and the callbacks that it provides to shape requests and responses.

HTTP Helper (api_call.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# lib/calculon/support/api_call.ex
defmodule Calculon.Support.APICall do
  use HTTPoison.Base

  # prepend the url with the server api route
  def process_url(url) do
    api_url <> url
  end

  # try to decode response bodies as JSON
  #   but reply with the raw body if there are
  #   any errors (e.g., invalid JSON)
  def process_response_body(body) do
    try do
      Poison.decode!(body, keys: :atoms!)
    rescue
      _ -> body
    end
  end

  # always convert the request body to JSON
  def process_request_body(body) do
    Poison.encode!(body)
  end

  # make sure we're posting JSON
  def process_request_headers(headers) do
    [{'content-type', 'application/json'} | headers]
  end

  # API url helper - will work in any env
  defp api_url do
    endpoint_config = Application.get_env(:calculon, Calculon.Endpoint)
    host = Keyword.get(endpoint_config, :url) |> Keyword.get(:host)
    port = Keyword.get(endpoint_config, :http) |> Keyword.get(:port)

    "http://#{host}:#{port}/api/v1"
  end
end

Now we can launch our app (mix phoenix.server) and in another terminal do things like this:

1
2
3
4
5
6
7
iex(1)> Calculon.Support.APICall.post!("/calculations", %{calculation: %{input: "1 + 1"}})
%HTTPoison.Response{body: %{"ok" => true},
headers: %{"cache-control" => "max-age=0, private, must-revalidate",
"connection" => "keep-alive", "content-length" => "11",
"content-type" => "application/json; charset=utf-8",
"date" => "Sun, 19 Apr 2015 20:24:49 GMT", "server" => "Cowboy",
   "x-request-id" => "DkMB98DVLHafB7eWdXoS"}, status_code: 201}

Note: By putting this code in lib/calculon/support, it will be available in all environments (test, development, production). This makes it useful for manual testing in development mode. Alternatively, we could put it in test/support, but then we could only use it from within our test suite.

Launching the server during tests

If we use the APICall helper in our test code at this point, we’ll get an error: ** (HTTPoison.Error) :econnrefused. That’s because Phoenix does not enable the server portion of the application when it’s run in the test environment. Fortunately, it’s not too hard to automate turning it on.

Let’s add a test helper to launch the API when we need it. We’ll put this code in the test/support directory because we don’t really need it outside of our test code.

API Helper (helpers.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
# test/support/helpers.ex
defmodule Calculon.Support.Helpers do
  def launch_api do
    # set up config for serving
    endpoint_config =
      Application.get_env(:calculon, Calculon.Endpoint)
      |> Keyword.put(:server, true)
    :ok = Application.put_env(:calculon, Calculon.Endpoint, endpoint_config)

    # restart our application with serving enabled
    :ok = Application.stop(:calculon)
    :ok = Application.start(:calculon)
  end
end

Writing integration tests

We have all of the pieces now. Let’s write a test.

Simple integration test (calculations_integration_test.exs) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# test/integration/calculations_integration_test.exs
defmodule CalculationsIntegrationTest do
  use ExUnit.Case

  # alias for convenience
  alias Calculon.Support.APICall

  # use the setup_all hook to make sure the API is
  # serving content during tests
  setup_all do
    Calculon.Support.Helpers.launch_api
  end

  test "POST /api/v1/calculations returns HTTP 201" do
    response = APICall.post!("/calculations", %{calculation: %{input: "1 + 1"}})
    assert response.status_code == 201
  end
end

We use the setup_all ExUnit callback to execute our helper that enables the API endpoints, and our HTTPoison helper to make the actual request. The HTTPoison response object has a status_code property that we can use to validate the HTTP status code returned by the server.

The test should pass.

1
2
3
4
5
6
7
$ mix test test/integration
.

Finished in 0.2 seconds (0.1s on load, 0.1s on tests)
1 tests, 0 failures

Randomized with seed 170902

Now let’s add some actual calculator functionality to Calculon using TDD. We’ll update our previous integration test to test for simple addition.

Updated integration test (updated_calculations_integration_test.exs) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
# test/integration/calculations_integration_test.exs
defmodule CalculationsIntegrationTest do
  use ExUnit.Case

  # alias for convenience
  alias Calculon.Support.APICall

  # use the setup_all hook to make sure the API is
  # serving content during tests
  setup_all do
    Calculon.Support.Helpers.launch_api
  end

  test "POST /api/v1/calculations returns HTTP 201" do
    response = APICall.post!("/calculations", %{calculation: %{input: "1 + 1"}})
    assert response.status_code == 201
  end

  test "simple addition" do
    request = %{calculation: %{input: "1 + 1"}}
    response = APICall.post!("/calculations", request)

    expected_response = %{calculation: %{input: "1 + 1", output: "2"}}

    assert response.status_code == 201
    assert response.body == expected_response
  end
end

This test should fail because we haven’t built the corresponding functionality yet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ mix test test/integration


1) test simple addition (CalculationsIntegrationTest)
test/integration/calculations_integration_test.exs:19
Assertion with == failed
code: response.body() == expected_response
lhs:  %{ok: true}
rhs:  %{calculation: %{input: "1 + 1", output: "2"}}
stacktrace:
test/integration/calculations_integration_test.exs:26

.

Finished in 0.3 seconds (0.1s on load, 0.1s on tests)
2 tests, 1 failures

Now we can update our controller to actually calculate output.

1
2
3
4
5
6
7
8
def create(conn, params) do
  input = params["calculation"]["input"]
  output = Calculon.Calculator.eval(input)

  conn
  |> put_status(201)
  |> json %{calculation: %{input: input, output: "#{output}"}}
end

The implementation of Calculon.Calculator.eval is included below. It’s similar in functionality to Code.eval_string/3, which evaluates arbitrary Elixir code strings, but it has heavy restrictions on the input for reasons of safety.

Now the tests pass:

1
2
3
4
5
6
7
$ mix test test/integration
..

Finished in 0.2 seconds (0.1s on load, 0.1s on tests)
2 tests, 0 failures

Randomized with seed 777178

The controller implementation doesn’t do any error checking, so let’s add a test for that.

1
2
3
4
5
6
# in test/integration/calculations_integration_test.exs
test "invalid input returns HTTP 400" do
  response = APICall.post!("/calculations", %{calculation: %{input:
""}})
  assert response.status_code == 400
end

This test fails because the controller responds with an HTTP status code of 201 even though it failed to perform any calculation. We’d like to let an API consumer know that they are providing invalid input, so let’s refactor the controller to return a 400 error if the calculator returns nil.

Controller with some error handling (calculations_controller.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# web/controllers/calculations_controller.ex
defmodule Calculon.CalculationsController do
  use Phoenix.Controller

  plug :action

  def create(conn, params) do
    input = params["calculation"]["input"]
    output = Calculon.Calculator.eval(input)

    if output do
      conn
      |> put_status(201)
      |> json %{calculation: %{input: input, output: "#{output}"}}
    else
      conn
      |> put_status(400)
      |> json %{error: %{input: ["unable to parse"]}}
    end
  end
end

Now our tests pass again:

1
2
3
4
5
6
7
$ mix test test/integration
...

Finished in 0.3 seconds (0.1s on load, 0.1s on tests)
3 tests, 0 failures

Randomized with seed 826740

Conclusion

These are the basic building blocks that you need in order to start doing integration testing for your Phoenix app. It mostly boils down to using something like HTTPoison to automate your HTTP requests and adding a setup callback that ensures your application is serving content during testing. The code for Calculon is on my github: https://github.com/dantswain/calculon.

Phoenix v0.10.0 and earlier

In Phoenix v0.10.0 and before, you may need to add the following to your launch_api helper:

1
Application.put_env :phoenix, :serve_endpoints, true

I’ve gotten it to work with v0.9.0 and v0.10.0. If you have trouble, feel free to leave a comment below and I’ll try to help.

Calculator code

This is by no means bullet proof, or even probably “good”. The basic idea is to parse the code into an AST and then check the form of the AST to make sure that the input is only doing one of a small list of allowed operations. My initial thought when I started writing this was to just use Code.eval_string/3, with the massive caveat that that would be very unsafe. I just couldn’t do it.

In theory, this could be extended with recursion to allow more complex operations. This is also might not be a great solution for a production app because the parsing and validation might be relatively heavy on the CPU.

Simplistic AST arithmetic parser (calculator.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
defmodule Calculon.Calculator do
  def eval(string) do
    ast = Code.string_to_quoted!(string)
    if is_simple_arithmetic?(ast) do
      {result, []} = Code.eval_quoted(ast)
      result
    else
      nil
    end
  end

  def is_simple_arithmetic?({op, _, args}) do
    Enum.member?([:+, :-, :*, :/], op) && bare_args?(args)
  end
  def is_simple_arithmetic?(_) do
  end

  def bare_args?([arg1, arg2]) when is_number(arg1) and is_number(arg2) do
    true
  end
  def bare_args?(_) do
    false
  end
end

Storing State in Elixir With Processes

- - posted in elixir

I’ve been getting into Elixir lately. I’m still learning, and one thing that helps me learn new languages is to take a problem I’ve solved in another language and solve it in the new language. A few months ago I wrote about storing state in erlang with processes and I thought it would be an interesting exercise to convert that discussion to Elixir.

I’ll start out with a solution that’s too simple to be useful. Then, by adding functionality and refactoring, show that a design pattern emerges that can be abstracted away from our implementation details. In this case, that pattern is a use-case for GenServer or Agent, and I’ll show how we can solve this problem using those tools.

State

State is how we make programs do non-trivial things. A video game’s states might include the position of the player, or a count of the number of bad guys that have been defeated. In most programming languages, we could count dead badguys by assigning the count to a variable and then updating that value each time another bad guy is defeated. Elixir, however, doesn’t allow us to keep around the value of a variable without actually carrying around a reference to that variable, which would get very cumbersome. So how do we keep track of state in Elixir? The answer is by using processes.

State is easy in object-oriented languages

We’ll use an example of a counter throughout the rest of this post. By a counter, I just mean a simple piece of code that’s capable of keeping a count value and provides an API to increment and/or access the value of the count. This is a very simple example of state; the state of the counter is the value of the count.

Object-oriented languages make it really easy and natural to store state in class instance variables. For example, to implement a simple counter in Ruby we could do something like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# a simple object-oriented counter in Ruby
class Counter
  def initialize
    @count = 0
  end

  def click
    @count += 1
  end
end

c = Counter.new
c.click # => @count is now 1
c.click # => @count is now 2

@count is an instance variable of the Counter class. When we create a new instance of the Counter class by calling Counter.new, @count gets initialized to a value of 0 in the constructor (initialize). Each time we call the click method, the value of @count is incremented by 1. @count holds the state of the counter. We can easily create multiple counters, each with its own independent state.

1
2
3
4
5
c1 = Counter.new
c2 = Counter.new
c1.click # => 1
c1.click # => 2
c2.click # => 1

In Elixir, we COULD do the following.

1
2
3
4
5
6
7
8
9
10
11
12
13
defmodule Counter do
  def new do
    0
  end

  def click(counter) do
    counter += 1
  end
end

c = Counter.new       # c = 0
c = Counter.click(c)  # c = 1
c = Counter.click(c)  # c = 2

That is, we could just keep using the count value. This has some obvious limitations. What if we want more than one process to be able to increment the counter? Each process would have to know the counter’s value, and keeping them all consistent would be very cumbersome.

Recursion and state

So how do we keep state in Elixir? We can get part of the way there using recursion by passing state from the intial function call to the next and so on. Here is an example of a counter in Elixir that prints out the value as it increments with each recursive function call.

Recursive counter (loop_counter.ex) download
1
2
3
4
5
6
7
8
9
10
defmodule LoopCounter do
  def go, do: go(0)

  defp go(n) do
    IO.puts("n is #{n}")
    # just to make sure this doesn't blow up our terminal
    :timer.sleep(1000)
    go(n + 1)
  end
end

Running this code from the iex shell will take over the shell because the function LoopCounter.go never returns. You can either exit the shell with the usual Control-g q or kill the job and connect to a new local shell with Control-g followed by k (kill the job), s (start a new shell), c (connect to the new shell).

1
2
3
4
5
6
7
8
9
10
11
12
iex(1)> c("loop_counter.ex")
[LoopCounter]
iex(2)> LoopCounter.go
n is 0
n is 1
n is 2
n is 3

User switch command
--> k
--> s
--> c

In LoopCounter, we use the input parameter n to the go/1 function to keep track of our state, and we increment it by recursively calling the same function with n + 1 as the parameter for the next call. The fact that we’re able to increment the value means that we are keeping track of the counter’s state. It’s just not very useful at this point because the recursive call loop completely takes over the current Elixir process.

We could use Kernel.spawn/1 to break our counter loop away from the main process (our shell) and Process.exit/2 to stop it.

1
2
3
4
5
6
7
8
9
10
11
12
iex(1)> pid = spawn(&LoopCounter.go/0)
n is 0
#PID<0.55.0>
n is 1
n is 2
n is 3
n is 4
n is 5
n is 6
n is 7
iex(2)> Process.exit(pid, :exit)
true

We’ve got control of our shell back, but we still have no way to control the counter or to get at its current value programmatically.

Controlling and querying state with messages

We use messages to communicate between processes in Elixir. The Elixir Getting Started guide’s section on processes does a good job of explaining message handling in Elixir (and even has a section that’s similar to this post).

Here is an example that adds a click message to our counter. We can use the click message to increment the counter’s value and send the updated value back to the calling process as a return message.

Simple signal-based counter (signal_counter.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
defmodule SignalCounter do
  def go do
    loop(0)
  end

  defp loop(n) do
    receive do
      {:click, from} ->
        send(from, n + 1)
        loop(n + 1)
    end
  end
end

We can create a counter using spawn and send messages to it. The counter expects messages to be tuples: {:click, from} where from should be the pid of the caller so we can send back the state as a return message.

1
2
3
4
5
6
7
8
9
10
11
iex(1)> c("signal_counter.ex")
[SignalCounter]
iex(2)> pid = spawn(&SignalCounter.go/0)
#PID<0.61.0>
iex(3)> send(pid, {:click, self})
{:click, #PID<0.53.0>}
iex(4)> receive do x -> x end
1
iex(5)> send(pid, {:click, self})
{:click, #PID<0.53.0>}
iex(6)> receive do x -> x end

This is still a little messy because the caller (in this case, us via the shell) has to know the message semantics and perform all of the message receiving. We can simplify this quite a bit by including the message handling in the module and providing a simple API to the outside world. Here’s a modified counter that does this, as well as adding a few new messages.

Signal-based counter with API layer (counter.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
defmodule Counter do
  # API methods
  def new do
    spawn fn -> loop(0) end
  end

  def set(pid, value) do
    send(pid, {:set, value, self()})
    receive do x -> x end
  end

  def click(pid) do
    send(pid, {:click, self()})
    receive do x -> x end
  end

  def get(pid) do
    send(pid, {:get, self()})
    receive do x -> x end
  end

  # Counter implementation
  defp loop(n) do
    receive do
      {:click, from} ->
        send(from, n + 1)
        loop(n + 1)
      {:get, from} ->
        send(from, n)
        loop(n)
      {:set, value, from} ->
        send(from, :ok)
        loop(value)
    end
  end
end

This makes the API much more intuitive by adding methods like Counter.click/0. Note that Counter.new/0 returns the pid of the underlying process and that we pass this pid to subsequent function calls. That’s because we’re using that process to hold the state of the counter! This is a very common Erlang and Elixir idiom.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
iex(1)> c("counter.ex")
[Counter]
iex(2)> c = Counter.new
#PID<0.62.0>
iex(3)> Counter.click(c)
1
iex(4)> Counter.set(c, 42)
:ok
iex(5)> Counter.get(c)
42
iex(6)> c2 = Counter.new
#PID<0.67.0>
iex(7)> Counter.get(c2)
0

Note that the Counter API functions use self/0 to get the PID of the caller. In Elixir, self/0 always returns the pid of the current process (it’s actually shorthand for Kernel.self/0). In the case of Counter.click/0, self/0 will be the pid of the process that called the function (the shell in the example above), because Counter.click/0 is just a simple function and will therefore be executed by the process that calls it.

Refactoring shared code and logical groups

There’s a lot of repeated code in the counter example above. Also, the implementataion of the counter logic is interspersed with lower-level message handling. You’re a good programmer and this rubs you the wrong way. So let’s refactor the code to share some of the common bits and clean it up quite a lot.

Refactored counter (gen_counter.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
defmodule GenCounter do
  # API
  def new do
    spawn(fn -> loop(0) end)
  end

  def click(pid) do
    make_call(pid, :click)
  end

  def get(pid) do
    make_call(pid, :get)
  end

  def set(pid, new_value) do
    make_call(pid, {:set, new_value})
  end

  # message handlers
  # handle_msg(message, current_state) -> {reply, new_state}
  defp handle_msg(:click, n), do: {n + 1, n + 1}
  defp handle_msg(:get, n), do: {n, n}
  defp handle_msg({:set, new_value}, _n), do: {:ok, new_value}

  # main state loop
  defp loop(state) do
    receive do
      {from, msg} ->
        {reply, new_state} = handle_msg(msg, state)
        send(from, reply)
        loop(new_state)
    end
  end

  # call helper
  defp make_call(pid, msg) do
    send(pid, {self(), msg})
    receive do x -> x end
  end
end

This code has the same functionality as before, but is factored to avoid repeated code and to hopefully group together the important bits of our API and logic. Our API consists of using the make_call/2 helper to send messages and receive the response, which effectively translates our API into message semantics. All of the actual counter logic boils down to defining how the process should react to various messages, which we implement in the handle_msg/2 callbacks.

A pattern is starting to emerge here. We have an API that makes calls by sending messages to an underlying process state loop, and that process state loop handles those messages by delegating to message callbacks. The loop/1 and make_call/2 helpers are low-level and generic. They also don’t handle a ton of edge cases: what happens if the process dies or takes too long to respond to a message? What happens when we send an unexpected message? What if we don’t need to respond to a message (i.e., a broadcast)? How does the state loop handle shutdown? What if we want our process not to respond immediately but instead do some work and then respond when that work is done?

Production-ready with GenServer

GenServer implements the generic part of our counter process, and does it very well. GenServer is an Elixir wrapper around the Erlang gen_server module, which is part of the OTP library.

Rewriting our counter using GenServer is fairly straightforward. Essentially, we add use GenServer to import the GenServer behavior, use GenServer.start_link/3 instead of Kernel.spawn/1, GenServer.call/3 instead of our make_call/2 helper, and implement init and handle_call callbacks to define our logic.

GenServer-based counter (counter_server.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
defmodule CounterServer do
  use GenServer

  # API
  def new do
    GenServer.start_link(__MODULE__, 0)
  end

  def click(pid) do
    GenServer.call(pid, :click)
  end

  def set(pid, new_value) do
    GenServer.call(pid, {:set, new_value})
  end

  def get(pid) do
    GenServer.call(pid, :get)
  end

  # GenServer callbacks

  # init(arguments) -> {:ok, state}
  # see http://elixir-lang.org/docs/v1.0/elixir/GenServer.html
  def init(n) do
    {:ok, n}
  end

  # handle_call(message, from_pid, state) -> {:reply, response, new_state}
  # see http://elixir-lang.org/docs/v1.0/elixir/GenServer.html
  def handle_call(:click, _from, n) do
    {:reply, n + 1, n + 1}
  end
  def handle_call(:get, _from, n) do
    {:reply, n, n}
  end
  def handle_call({:set, new_value}, _from, _n) do
    {:reply, :ok, new_value}
  end
end

By using GenServer, we get a ton of functionality for free. For example, GenServer.call/3 takes an optional timeout argument which we can use to guarantee that the calling process never hangs while waiting for the GenServer to respond. We haven’t implemented it here, but GenServer can also handle termination callbacks, broadcast messages, bare messages (via Kernel.send/2 as opposed to GenServer.call/3 or GenServer.cast/2), process registration, idle timeouts, and even hot code swapping.

An Elixir bonus - using Agent

GenServer is essentially a direct adaptation of Erlang’s gen_server. The Elixir developers realized that many times we use a fairly small subset of GenServer’s capabilities, more or less just to store some state and provide an API for accessing that state. The Agent module is designed for just that use case.

Here is how we could rewrite our counter using the Agent module.

Agent-based counter (counter_agent.ex) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
defmodule CounterAgent do
  def new do
    Agent.start_link(fn -> 0 end)
  end

  def click(pid) do
    Agent.get_and_update(pid, fn(n) -> {n + 1, n + 1} end)
  end

  def set(pid, new_value) do
    Agent.update(pid, fn(_n) -> new_value end)
  end

  def get(pid) do
    Agent.get(pid, fn(n) -> n end)
  end
end

This reduces our Counter to just four named functions. Much of the work is done by anonymous functions that we supply to the various Agent functions. For example, the initializer is simply fn -> 0 end to set the initial value to zero, and the getter is fn(n) -> n end which just returns the current state.

The end

There it is. We use processes to keep track of state in Elixir, and messages between processes to access and control that state. Elixir is very similar to, and in fact 100% compatible with, Erlang in this way. Elixir provides the Agent module to handle many of the most basic use cases, and the GenServer module to provide more flexibility.

Solving problems by using processes and messaging can be hard to wrap your head around at first. However, once it clicks for you, many problems become much easier to solve in this paradigm.

Storing State in Erlang With Processes

- - posted in erlang

State is how we make programs do non-trivial things. If we’re writing a video game, one of its states might be a count of the number of bad guys that have been defeated. In most programming languages, we would accomplish this by assigning the count to a variable and then updating the value each time a new bad guy is defeated. Erlang, however, doesn’t allow us to change the value of a variable once it has been set (single assignment). So how do we keep track of state in Erlang if we’re not able to change the value of a variable? The answer is by using processes.

Single Assignment

Erlang derives a lot of its power from single assignment. Single assignment means that once a value is assigned to a variable, no other value may be assigned to the same variable. You’ll see something like the following in most tutorials on Erlang.

1
2
3
4
1> X = 1.
1
2> X = 2.
** exception error: no match of right hand side value 2

In the first step, Erlang tries to pattern match X with the value 1. Because X is unbound, it gets bound to the value 1. From now on, when X is the left-hand side of the = pattern matching operator, normal pattern matching will happen. That’s why we get an error in the second step - Erlang is trying to match X, which has a value of 1, with the value 2 therefore causing a no match error. We can still match X as long as there is a corresponding 1 on the right-hand side, as in the following.

1
2
3
4
3> X = 1.
1
4> {X, 2} = {1, 2}.
{1,2}

Single assignment is a huge benefit in Erlang because it reduces the possibility for function calls to have side effects. Side effects can make writing highly concurrent code very difficult.

State

We’ll use an example of a counter throughout the rest of this post. By a counter, I just mean a simple piece of code that’s capable of keeping a count value and provides an API to increment and/or access the value of the count. This is a very simple example of state; the state of the counter is the value of the count.

Object-oriented languages make it really easy and natural to store state in class instance variables. For example, to implement a simple counter in Ruby we could do something like this.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# a simple object-oriented counter in Ruby
class Counter
  def initialize
    @count = 0
  end

  def click
    @count += 1
  end
end

c = Counter.new
c.click # => @count is now 1
c.click # => @count is now 2

@count is an instance variable of the Counter class. When we create a new instance of the Counter class by calling Counter.new, @count gets initialized to a value of 0 in the constructor (initialize). Each time we call the click method, the value of @count is incremented by 1. @count holds the state of the counter. We can easily create multiple counters, each with their own independent state.

1
2
3
4
5
c1 = Counter.new
c2 = Counter.new
c1.click # => 1
c1.click # => 2
c2.click # => 1

Recursion and state

So how do we keep state in Erlang? We can get part of the way there using recursion by passing state from the intial function call to the next and so on. Here is an example of a counter in erlang that prints out the value as it increments with each recursive function call.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
%%% loop_counter.erl
% a recursive loop counter
%  takes over the current process, and no access to the value
-module(loop_counter).
-export([go/0]).

go() ->
  go(0).

go(N) ->
  io:format("N is ~p~n", [N]),
  % just to make sure this doesn't blow up our terminal
  timer:sleep(1000),
  go(N + 1).

%%% Erlang shell
1> c(loop_counter).
{ok,loop_counter}
2> loop_counter:go().
N is 0
N is 1
N is 2
N is 3

(hit Ctrl-c)
BREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded
       (v)ersion (k)ill (D)b-tables (d)istribution

In this example, we use the input parameter N to the go/1 function to keep track of our state, and we increment it by recursively calling the same function with N + 1 as the parameter for the next call. The fact that we’re able to increment the value means that we are keeping track of the counter’s state. It’s just not very useful at this point because the recursive call loop completely takes over the current erlang process.

We can use erlang:spawn/1 to break our counter loop away from the main process (our shell).

1
2
3
4
5
6
7
8
9
10
11
1> c(loop_counter).
{ok,loop_counter}
2> Pid = spawn(fun loop_counter:go/0).
N is 0
<0.39.0>
N is 1
N is 2
N is 3
N is 4
3> q().
ok

We’ve got control of our shell back, but now we have no way to control the counter or to get at its current value programmatically.

Controlling and querying state with messages

In the previous section we created a counter that increments its own value by calling itself recursively, but we had no way to control the counter or access its state from any external code. We can use messages to do this. Here is an example that adds a click message to our counter. We can use the click message to increment the counter’s value and send the updated value back to the calling process as a return message.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
%%% counter.erl
% spawns a counter loop that listens for a click message
-module(counter).
-export([new/0, click/1]).

% spawn a counter, returning the pid
new() ->
  spawn(fun() -> loop(0) end).

% recursive loop with receive block
loop(N) ->
  receive
    {click, From} ->
      From ! N + 1,
      loop(N + 1)
  end.

% API function to increment a counter given its pid
click(Pid) ->
  Pid ! {click, self()},
  receive V -> V end.

%%% in Erlang shell
1> c(counter).
{ok,counter}
2> C = counter:new().
<0.39.0>
3> counter:click(C).
1
4> counter:click(C).
2
5> counter:click(C).
3

When we call counter:new/0, the function spawns a new recursive loop with a zero initial counter value and returns the pid of the spawned process. The loop function immediately enters a receive block, where it waits indefinitely for messages from any process. We’ve set it up to only listen for one message: {click, From}, with the expectation that From is the pid of the calling process.

Here is an example of creating a counter process from the terminal and then directly sending a message to it and waiting to receive a message back.

1
2
3
4
5
6
7
8
1> c(counter).
{ok,counter}
2> C = counter:new().
<0.39.0>
3> C ! {click, self()}.
{click,<0.32.0>}
4> receive V -> V end.
1

This is a little cumbersome. We don’t want to have to write the message call and the receive block each time we want to increment the counter. That would also be unsafe because it leaves a lot of room for error. Thus, we introduce the API method counter:click/1 (see the code above) that takes a counter pid and knows the correct way to send and receive a click message.

Note that we use self/0 in the counter:click/1 call to automatically get the pid of the calling process. This works because counter:click/1 is being evaluated by the calling process. If we had a call to self/0 in the counter:loop/1 function, it would return the value of the spawned process that is evaluating the loop.

Using this paradigm, it’s easy to create and control multiple counters by spawning multiple processes. Moreover, the API hides most of the underlying implementation details and we can work with the counter instances without having to know that their value is actually a process identifier.

1
2
3
4
5
6
7
8
9
10
11
12
1> c(counter).
{ok,counter}
2> C1 = counter:new().
<0.39.0>
3> C2 = counter:new().
<0.41.0>
4> counter:click(C1).
1
5> counter:click(C1).
2
6> counter:click(C2).
1

We could also modify the loop function to listen for a set message to allow us to set the value of the counter manually.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
% adding a message handler to set the value of the counter
loop(N) ->
  receive
    {click, From} ->
      From ! N + 1,
      loop(N + 1);
    {set, Value, From} ->
      From ! ok,
      loop(Value)
  end.

% API function to set the value of a counter, given its pid
set(Pid, Value) ->
  Pid ! {set, self(), Value},
  receive V -> V end.

Production-ready using gen_server

Our counter from above has a lot of potential issues. What happens when a counter process recieves an unexpected message? What happens when the system gets busy and it is unable respond to a click message in a timely manner? How do we stop the counter and free up any system resources that it is using? The code also gets unmanageable quickly as we add more and more messages.

Erlang comes with a set of behavior interfaces that solve the kinds of problems that one generally comes across when trying to solve problems using Erlang. This set of modules is called OTP (formerly the “open telecom platform”, now just “OTP”).

The OTP gen_server behavior is designed and implemented to implement a design pattern that solves the problems that we’ve been exploring so far in this post. It implements a stateful spawned process behavior in a robust and complete way, and solves many problems that we haven’t even thought about yet.

I won’t go into a lot of depth here because the point of this post is not to teach you about behaviors or proper OTP design. If you want more detail, see the gen_server man page and the Learn You Some Erlang Clients and Servers chapter. However, the following code should give you a decent idea of how a gen_server solves the problem of keeping state in Erlang.

Here is an implementation of a simple counter using gen_server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
%%% counter_server.erl
% simple counter implemented as a gen_server
-module(counter_server).
-behavior(gen_server).

% API
-export([new/0, click/1]).

% required by gen_server
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).

%%% API methods
new() ->
  gen_server:start(?MODULE, [], []).

click(Pid) ->
  gen_server:call(Pid, click).

%%% gen_server callbacks
%%%   these are required to implement the gen_server behavior
%%%   we're really only using init and handle_call
init([]) ->
  % the second value is the initial counter state
  {ok, 0}.

handle_call(click, _From, N) ->
  % the second value is sent back to the caller
  % the third value is the new state
  {reply, N + 1, N + 1}.

% basically, we ignore these, but keep the same counter state
handle_cast(_Msg, N) ->
  {noreply, N}.
handle_info(_Msg, N) ->
  {noreply, N}.
code_change(_OldVsn, N, _Other) ->
  {ok, N}.
terminate(_Reason, _N) ->
  ok.

%%% Erlang console
1> c(counter_server).
{ok,counter_server}
2> {ok, C} = counter_server:new().
{ok,<0.39.0>}
3> counter_server:click(C).
1
4> counter_server:click(C).
2
5> counter_server:click(C).
3

This may seem like a lot of code to solve a simple problem, but we’ve gained a ton of flexibility just by implementing within this framework. There are also plenty of templates available to produce all of the boilerplate code for you (in Emacs, you can use M-x tempo-template-erlang-generic-server from the Erlang package).

As an example, adding a set/2 method similar to the one above is just a matter of adding an API method and a handle_call callback.

1
2
3
4
5
6
7
8
9
10
11
12
13
%%% API
set(Value, Pid) ->
  gen_server:call(Pid, {set, Value}).

...

handle_call({set, Value}, _From, _N) ->
  {reply, ok, Value};

% existing callback from above, shown for reference
%  (the two have to appear together for pattern matching to work)
handle_call(click, _From, N) ->
  {reply, N + 1, N + 1}.

The end

So there you have it. We use processes to keep track of state in Erlang, and we use messages to access and control that state. In a lot of ways, processes in Erlang are similar to class instances in other languages. In many other ways, they are very different. Solving problems in Erlang takes a different way of thinking: thinking in terms of processes. This can be difficult to grok at first, but once you understand it, many problems are much more easily solved this way.

New Blog Design

- - posted in design, meta, robots

I think a lot of developers have this intention of creating and maintaining a blog. I started this one over a year ago and have published a whopping two posts. I’d call it a success!

After those two posts, I felt it was obviously time for a redesign. The main purpose of this post is to give credit to some people that contributed in one way or another to this redesign.

This site is powered by Octopress, which is a pretty cool framework for blogging. You get to write posts and pages in Markdown, which is a pretty pleasant way to write in plain text. Unfortunately, the default Octopress theme is… maybe not “ugly” per se, but certainly more utilitarian than attractive. The Octopress website is a good example. Fortunately, there are lots of themes for Octopress. I went with foxslide, by Adrian Artiles. Foxslide uses a great css library called Twitter Bootstrap.

I also want to give credit to the Princeton University President’s Lecture Series. I extracted the background image of this site from a poster that was prepared (presumably by their graphic designers) for a lecture by Naomi Leonard, my Ph.D. Advisor. There is a video of her talk on youtube; it was titled “Flocks and Fleets: Collective Motion in Nature and Robotics”. I helped prepare a couple of her slides.

The reason I chose this particular background is because it was constructed using a figure from my research. The figure shows the trajectories of a group of agents circling a common point with oscillatory speed. Part of my Ph.D. research was coming up with provably-stable control laws to coordinate such formations. That work was motivated by observations that banded killifish behave in a similar fashion. One benefit of motion with oscillatory speed is that the fish move relative to one another in a regular pattern. We showed that such motion can lead to faster decision making at the group level.

Erlang: First Element in a List That Matches Some Condition

- - posted in erlang

The short answer:

1
2
3
4
5
6
7
8
first(L, Condition, Default) ->
  case lists:dropwhile(fun(E) -> not Condition(E) end, L) of
    [] -> Default;
    [F | _] -> F
  end.

3 = first([0, 1, 2, 3, 4, 5], fun(E) -> E > 2 end, 0).
0 = first([0, 1, 2, 3, 4, 5], fun(E) -> E < 0 end, 0).

Thanks to Odobenus Rosmarus, who provided this solution to my Stack Overflow question.

The long answer:

I recently needed an Erlang function that looks through a list and returns the first element matching some condition, without looking through the rest of the list.

It’s relatively straightforward to use list comprehension to find all of the elements in a list that match some condition:

1
2
3
4
5>  L = [0, 1, 2, 3, 4, 5].
[0,1,2,3,4,5]
6> Matches = [ E || E <- L, E > 2].
[3,4,5]

And then we can get the first element using matching:

1
2
3
4
7> [First | _Rest] = Matches.
[3,4,5]
8> First.
3

Using this strategy, we have to evaluate the test condition (M > 2) for every element in the list before we can extract the first one. That’s very inefficient — what if the list was very long?

Here’s a solution I came up with that uses recursion:

1
2
3
4
5
6
first([E | Rest], Condition, Default) ->
  case Condition(E) of
    true -> E;
    false -> first(Rest, Condition, Default)
  end;
first([], _Cond, Default) -> Default.

This function returns a match immediately when it is found, it lets us provide the condition as a function, and it returns a default argument if no match is found. We can verify that it stops evaluting by having the Condition function tell us when it’s called:

1
2
3
4
5
5> first([1, 2, 3, 4, 5], fun(E) -> io:format("Evaluating for ~p~n", [E]), E > 2 end, 0).
Evaluating for 1
Evaluating for 2
Evaluating for 3
3

I posted this as a question on Stack Overflow and Odobenus Rosmarus suggested a more elegant solution. I wrapped that solution into a more general version:

1
2
3
4
5
first(L, Condition, Default) ->
  case lists:dropwhile(fun(E) -> not Condition(E) end, L) of
    [] -> Default;
    [F | _] -> F
  end.

This solution uses the lists:dropwhile/2 function to stop evaluation when a match is found. I wasn’t sure that was what it would do, so I verified using the same technique above.

One thing I’m still not sure about is how these solutions compare in terms of memory usage, especially for very large lists.

Opening a New Instance of Aquamacs From the Command Line

- - posted in bash, open, os x, shell scripts

The short answer:

1
2
#!/bin/bash
open -na Aquamacs $*

The long answer:

If you’ve used the OS X command line for any length of time, you’ve probably (hopefully) come across the open command. Its job is (shockingly enough) opening files and applications using Finder. It works for files:

1
$ open index.html

opens index.html in your browser. It also works for applications:

1
$ open -a Mail

opens Mail.app. It also works for opening files with an application:

1
$ open -a Aquamacs config/routes.rb

opens config/routes.rb in Aquamacs.