Clementson's Blog

Test Libraries for Erlang

Wednesday, June 13, 2007

Recently, on the Erlang mailing list, Joel Reymont asked about how one specifies tests in Erlang. Ulf Wiger posted a good summary of the various library options that are available:

1. Erlang/OTP test server: This is what OTP uses, and they have several thousand test cases written for OTP. It's used by several large commercial projects. It has support for multi-target testing, and automated testing of embedded systems.
2. eUnit: This is for simpler unit testing, and can surely be combined with the Test Server.
3. Erlang QuickCheck: A commercial testing tool.

"That's a matter of interpretation, I think.

While I agree that QC is the only tool that lets you lay down the rules for testing rather than describing each individual test, even plain test cases in Erlang are declarative by nature.

For example, when using the OTP test server, you write down test cases as individual functions (one test case - one function), and the test succeeds if the function returns ok; everything else is considered a failure. The test case itself usually relies heavily on (declarative) pattern matching. An example:

rdbms_unique_ix(doc) ->
    "Test the 'unique' option for indexes";
rdbms_unique_ix(suite) ->
  [];   % there are no sub-tests
rdbms_unique_ix(Config) when is_list(Config) ->
    ?line {atomic,ok} = rdbms:create_table(
                          u_ix,
                          [{ram_copies, [node()]},
                           {attributes, [key, value]},
                           {rdbms, 
                            [
                             {indexes, 
                              [{{value,value},?MODULE,ix_value,[],
                                [{type,set},
                                 {unique, true}]}]}
                            ]}
                          ]),
    ?line trans(fun() ->
                        mnesia:write({u_ix,1,a}),
                        mnesia:write({u_ix,2,b})
                end),
    ?line trans(fun() ->
                        [{u_ix,1,a}] = 
                            mnesia:index_read(u_ix,a,{value,value}),
                        [{u_ix,2,b}] = 
                            mnesia:index_read(u_ix,b,{value,value})
                end),
    ?line fail_trans(fun() ->
                             mnesia:write({u_ix,3,a})
                     end).

The ?line macros are optional, and are simply utility macros to help identify exactly where the test went wrong. The helper functions trans/1 and fail_trans/1 will exit if passing the argument as a fun to mnesia:transaction/1 commits or aborts respectively, thus acting as assertions.

This is actually quite similar to how one would go about writing QC tests, but with the main difference that you work with symbolic input, and then let QC generate loads of different input values.

Here's an example using eUnit:

-module(fib).
-export([fib/1]).
-include_lib("eunit/include/eunit.hrl").

fib(0) -> 1;
fib(1) -> 1;
fib(N) when N > 1 -> fib(N-1) + fib(N-2).

fib_test_() ->
    [?_assert(fib(0) == 1),
     ?_assert(fib(1) == 1),
     ?_assert(fib(2) == 2),
     ?_assert(fib(3) == 3),
     ?_assert(fib(4) == 5),
     ?_assert(fib(5) == 8),
     ?_assertException(error, function_clause, fib(-1)),
     ?_assert(fib(31) == 2178309)
    ].

It's also pretty declarative in that you focus on what is supposed to happen, but it's of course "pin-prick testing", compared to QC, which is much closer to model-checking, but much easier to get into.

If you really want to get into esoteric stuff (and I'm sure you do), you can look at some of the stuff done by Thomas Arts et al. In

http://www.ituniv.se/~arts/papers/fates04.pdf

they describe how they went about testing a leader election implementation, which had first been verified using model checking, then further tested using abstract traces (+ model checking), which revealed serious bugs; then the code was reimplemented and verified with abstract traces, but QuickCheck still found faults in the tested code.

If nothing else, it illustrates what a thankless job software testing is. (:

(If you want to explore model checking for Erlang, then McErlang ought to be someting to consider: http://babel.ls.fi.upm.es/~fred/McErlang/)"

"There are a number of tools you can use to test apart from Eunit. For instance:

• the Erlang Test Server for Unit/System Testing (and the test helper skeletons for emacs) --- http://www.erlang.org/project/test_server/index.html
• the Dialyzer for contract/type testing --- comes in recent Erlang/OTP releases
• Tsung/Tsunami for load testing --- http://tsung.erlang-projects.org/
• There is an additional commercial product for very advanced testing: http://www.quviq.com/
• You can wrap some of these tools up with a bash_script called erlang_automated_build that will perform them, e-mail on success or failure and graph the results daily. This is in the jungerl: http://jungerl.cvs.sourceforge.net/jungerl/jungerl/lib/erlang_automated_build/

Most of the above will install straight from the comprehensive Erlang Archive Network, CEAN: http://cean.process-one.net/

There are also tutorials etc: http://wiki.trapexit.org/index.php/Testing_with_Test_Helper http://forum.trapexit.org/viewtopic.php?t=6076"

"and also Yatsy: http://code.google.com/p/yatsy"

"Depends on the sort of tests you want to write.

The Erlang Test Server will allow you to write as complex system tests as you want. It runs through a set of suite and test set-up routines, you execute your tests and it tears down your environment.

Typically I create a new db schema, load it with test data, start an application (or applications) and then execute tests against the known environment - depending on the product.

For large web apps you can use Tsung to record and script the http input and run regression tests, or you can use it as a 'white noise' tester and fire a million random inputs in and then look for crash reports (both work for me)."

"I second this.

We use the Erlang Test Server for subsystem tests and system tests. In the systems we're building right now, we have some half a million lines of Erlang code (over a million lines of C), and about 70 different erlang applications, not counting the OTP apps. The Test Server is our main vehicle for stitching together automated tests, and our testers also write test suites for system-level tests, similar to those Gordon described. It works wonderfully."

"eUnit has a lower entry threshold. You can easily write a test suite for the Test Server that runs all your eUnit tests."

"> How does yatsy differ from test_server and EUnit?

It takes (almost) the same ETS SUITE files as input. The main difference to ETS is that it is available from a public repository and thus can be improved and maintained properly.

We (Kreditor) use it for our automatic testsuites both nightly builds as well as integrated with Cruise Control (i.e all test-suites are automatically run as soon as someone does a checkin into our svn repos).

We are also using EUnit which is great to use for a test-driven development aproach. See this the excellent blog entry for example:

http://pragdave.pragprog.com/pragdave/2007/04/testfirst_word_.html

> Integration with Yaws?

Yatsy can produce reports as HTML on disk as well as deliver them via the Yaws server that automatically is setup.

> But I do not get it - why do you need 2 different test frameworks?
> Does not Yatsy include functionality of EUnit?

Yes you can do similar kind of tests with Yatsy. However, EUnit is much more light-weight. We are using:

Yatsy - for system tests. It can execute on the target node or from a remote node. We test major parts of our system where we prepare DB contents, setup timeouts and config data. We run tests both within the target node as well as remotely via our Xml-Rpc interface.

EUnit - for unit tests (i.e in general development where appropriate). While you are developing code you define input and expected output for local code in the module and run the EUnit generated test/0 function until your assertions are fulfilled."