Living documentation with Dexy

In the current Sphinx documentation of cmake-boost-test, chapter Running the test executables explains how to use CTest. It shows how to register tests in the CMake list file that you can then execute via a simple invocation of "make test". It also shows the output of that call, that is, the output as it was when I wrote the chapter and manually copied it into the Sphinx source file. Since then I have added other tests, which I also describe in other parts of the documentation, but whose output does not appear in the example.

So how do we keep those examples up-to-date? It is a boring task that is easy to forget, or should I say, neglect. This is were the Python tool Dexy comes to the rescue. Dexy allows you to capture the output of any command and place that output in your documents. To get a feel of how it works, take a look at an earlier version of the Sphinx document running-tests.rst. It contained the following example output of "make test":

Running tests...
Test project /home/pieter/bitbucket.org/cmake-boost-test/build/beck
    Start 1: my-boost-test
1/4 Test #1: my-boost-test ....................   Passed    0.01 sec
    Start 2: my-static-boost-test
2/4 Test #2: my-static-boost-test .............   Passed    0.01 sec
    Start 3: my-static-name-boost-test
3/4 Test #3: my-static-name-boost-test ........   Passed    0.01 sec
    Start 4: my-dynamic-boost-test
4/4 Test #4: my-dynamic-boost-test ............   Passed    0.09 sec

100% tests passed, 0 tests failed out of 4

Total Test time (real) =   0.13 sec

As mentioned, that output does not correspond to the current output of "make test". With Dexy you replace that hard-coded example by the following Dexy construct:

{{ d['make-test.sh|bash']|indent(2) }}

When you run Dexy, it creates a copy of running-tests.rst where the above construct is replaced with output of the shell script make-test.sh [1], which it indents by 2 columns. In other words, it replaces the construct by:

Running tests...
Test project /home/pieter/bitbucket.org/cmake-boost-test/build/beck
    Start 1: my-boost-test
1/5 Test #1: my-boost-test .......................   Passed    0.12 sec
    Start 2: my-static-boost-test
2/5 Test #2: my-static-boost-test ................   Passed    0.11 sec
    Start 3: my-static-name-boost-test
3/5 Test #3: my-static-name-boost-test ...........   Passed    0.15 sec
    Start 4: my-static-find-package-boost-test
4/5 Test #4: my-static-find-package-boost-test ...   Passed    0.04 sec
    Start 5: my-dynamic-boost-test
5/5 Test #5: my-dynamic-boost-test ...............   Passed    0.07 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) =   0.53 sec

Dexy uses a config file that describes what it has to do. For this example the config file, named .dexy, could look like this:

{
  "running-tests.rst|jinja" {
    inputs: ["make-test.sh|bash"]
  },
}

This config gives an indication of how Dexy works. It passes the file running-tests.rst through a so-called filter with the name "jinja". This filter treats the file mentioned before it as a Jinja template and evaluates that template [2]. To be able to do so, Dexy first passes make-test.sh through the filter "bash", which executes the shell script and captures its output. The end result is a copy of running-tests.rst that contains the output of a new invocation of "make test".

The above barely scratches the surface of what Dexy can do and you should visit the Dexy website for some impressive examples. In the "Garlicsim" example Dexy runs a simulation in Python, analyzes the results in R and packages it in a single document that describes simulation, code and outcome.