Cram tests: dune's hidden gem

I'm a strong advocate of unit tests. I can confidently say they've saved me from introducing regressions countless times. Today I want to share one of the hidden gems in OCaml's testing story with dune: cram tests.

Cram tests are snapshots of interactive shell sessions. They offer a versatile framework for testing OCaml projects, and you can use them for any CLI.

I'm surprised this pattern isn't more common outside the (small) world of OCaml. It came from Python (bitheap.org/cram), and implementations exist in Go (https://github.com/mgeisler/cram) and other languages. Maybe cram tests never reached the JavaScript world (??), and they're a bit old.

A cram test file contains CLI invocations followed by the expected output from either stdout or stderr. The test runner, dune in this case, runs those invocations and diffs the actual output against the expected output.

For example:

Here there is just a coment about this test, since there is no space at the start
it's treated as a comment.
 
	$ echo "This command runs as part of the test" > data.txt
 
We want to see the content of "data.txt"
 
  $ cat data.txt
  This command runs as part of the test
 
The command `cat data.txt` gets diffed against the next line `  This command runs as part of the test
` from this file, and that's the first beauty of cram tests.

The syntax has three rules:

• Text without indentation is a comment
• Indented lines with a dollar sign run in the shell
• Lines after a shell command, indented and prefixed by >, contain the expected output from either stdout or stderr

cram test editor screenshot

This screenshot shows the syntax highlighting in my editor and a real .t file.

dune supports cram tests out of the box and fits into your existing dune test workflow.

With dune < 3, you need to enable them manually. Add (cram enabled) to dune-project, and create a dune-project file at the root of your project if you don't already have one. If you are using dune > 3, there is nothing to do because cram tests are already enabled.

To experience the workflow before using it in a real-world scenario, open your dune project and follow this tutorial.

Start with the simplest possible test.

1. Create a file called simple.t with:

  $ echo 'lola'

2. Run the test and let it crash in one of two ways:

• run the entire “test suite” with dune runtest
• run a specific test with its respective alias, since all cram tests are automatically dune aliases: dune build @simple —watch

The execution will produce this output:

File "tests/simple.t", line 1, characters 0-0:
diff --git a/_build/.sandbox/e72f0d9cb592f23848ccd62f4a630862/tests/simple.t b/_build/.sandbox/e72f0d9cb592f23848ccd62f4a630862/tests/simple.t.corrected
index de6f8c28..ef3231ef 100644
--- a/_build/.sandbox/e72f0d9cb592f23848ccd62f4a630862/tests/simple.t
+++ b/_build/.sandbox/e72f0d9cb592f23848ccd62f4a630862/tests/simple.t.corrected
@@ -1 +1,2 @@
   $ echo 'lola'
+  lola

3. Accept the result as correct. You can update the snapshot the boring and tedious way, either manually or automatically with promotion.

Run dune build @simple —auto-promote to update the simple.t file for you.

To ensure everything works, run the tests again. In this case, dune build @simple should pass with empty output. Empty in dune means very good.

Directories with .t and a run.t file inside are also valid cram tests.

Keeping all your tests as .t files can get out of hand pretty quickly. Some test suites could contain hundreds of tests, and some tests might contain fixtures or config files.

You might write a perfect test only to be greeted with [ERROR] Program "your_binary" not found!. This happens because dune doesn't automatically know which executables should be accessible during cram test execution.

There are four ways to expose your executable to cram tests:

• option A: Add a public_name to your executable stanza, such as (public_name amazing-binary). This makes the executable available to the test suite as amazing-binary.exe. When users install your package, the executable will be under my-opam-package.amazing-binary.
• option B: Make the executable part of the dune installation with the install stanza, (install (section bin) (files amazing-binary.exe)). Do not confuse dune installation with opam installation. Dune installation offers a way of "copying freshly built artifacts from the workspace to the system", while opam installs opam packages in your opam switch by downloading them from opam repositories, building from source, and so on.
• option C: Use (env _ (binaries ../bin/amazing-binary.exe)) to make an executable available as a binary. amazing-binary will be available to the test suite, and you can choose which environment will have it. This is useful if you don't want to rely on the executable's exact location or expose it to the user. (Thanks to sim642 for the tip).
• option D: Specify the executable directly as a dependency in your cram test with (deps %{exe:../bin/amazing-binary.exe}). The dune documentation has more information about the cram stanza.

If you're not sure which to choose, I recommend starting by setting a public_name in your executable stanza. It's the most explicit approach, and if you are working on a CLI, you will still need a public_name to expose it on the opam package.

Effective cram tests require more than copying command output. These principles will help you write robust and maintainable snapshot tests.

Your tests should be hermetic, meaning they're self-contained and don't depend on external state. Each test should clean up after itself, and running it multiple times should produce the same result. For example:

  $ mkdir -p tmp
  $ cd tmp
  $ echo "hello" > test.txt
  $ cat test.txt
  hello
  $ cd ..
  $ rm -rf tmp

Hardcoding absolute paths is a common pitfall. It makes tests brittle and non-portable. Dune provides several environment variables for writing path-independent tests:

DON'T do this
  $ cat /home/user/project/test.txt
 
DO this instead
  $ cat $DUNE_ROOT/test.txt

Some useful environment variables include:

• $DUNE_ROOT: Points to your project root
• $INSIDE_DUNE: Set when running inside dune's test suite
• $PWD: Current working directory
• $DUNE_BUILD_DIR: Path to dune's build directory

This more complete example shows the variables in action:

  $ echo $DUNE_ROOT | grep -o '[^/]*$'
  my-project
 
Create a file relative to project root
  $ cat > $DUNE_ROOT/input.ml <<EOF
  > let greeting = "Hello, World!"
  > let () = print_endline greeting
  > EOF
 
Test the compiled output
  $ dune exec ./bin/main.exe
  Hello, World!

This approach keeps your tests consistent across different environments and developer machines. It's especially important if you plan to run them in CI or other developers' local environments.

A good cram test should explain its purpose and any non-obvious setup. This makes it easier for others, or yourself in 6 months, to understand what's being tested and why. Comments should focus on the test's intent rather than describe the commands being run.

The adoption of cram tests by prominent projects in the OCaml ecosystem shows their effectiveness.

Dune itself uses cram tests extensively throughout its codebase, with over 800 tests covering everything from basic builds to complex workspace configurations. Dune has also used cram tests for bug reporting. When users encounter issues, they're encouraged to submit a minimal reproduction case as a cram test. This practice has been so successful that many bug reports in Dune's issue tracker include a .t file demonstrating the problem.

Melange is another great example. It uses blackbox tests to ensure that JavaScript compilation remains consistent: https://github.com/melange-re/melange/tree/main/test/blackbox-tests.

For reference, here is some of my work migrating to cram tests:

• Migrate ReasonML’s tests to cram suite: https://github.com/reasonml/reason/pull/2694
• styled-ppx contains a few suites of snapshots (mostly ppx transformations): https://github.com/davesnx/styled-ppx/tree/main/packages/ppx/test/snapshot
• reason-react-ppx keeps its ppx generation and JavaScript output consistent: https://github.com/reasonml/reason-react/tree/main/ppx/test

TLDR: cram tests are nice, actually.