Snapshot tests for your own ppx

Testing is crucial when building preprocessor extensions (ppx) in OCaml. You want to ensure your ppx works correctly and continues to work as you make changes. After experimenting with different approaches, I've found that cram tests fit the task well.

As I explained in my previous post, cram tests are essentially command-line snapshot test sessions.

They're simple text files containing commands and their expected output, which makes them particularly valuable for ppx development:

1. They act as living documentation, showing exactly how your ppx behaves in different scenarios
2. They make it easy to report bugs because users can simply share a cram test that demonstrates the issue
3. They provide a sandboxed environment that mimics real-world usage

I assume that if you're reading this, you're already convinced it's a good idea. If you had any doubts left, I count those three arguments as enough to close the case.

Let's get into the how-to. Once you have an opam switch, a dune project, and a ppx, you're almost ready to go. If you're missing any of those, check https://github.com/ml-in-barcelona/ppxlib-simple-example for a minimal setup.

This is unrelated to testing, but if you want to learn more about ppxes, check ppx-by-example.

The goal is to make our ppx transformations accessible to our tests through an executable. Before writing any tests, it helps to understand how Dune organizes build artifacts and executables.

Dune scans your project for dune files. When it finds one, it generates build artifacts in the _build directory. The contents of these dune files determine exactly what gets built.

For example, when you specify a ppx rewriter:

; a library that uses a ppx
(library
  (name my_cool_library)
  (preprocess (pps my-ppx)))

Dune looks for a library marked as a ppx rewriter. The public_name is what matters.

(library
  (name my_ppx)
  (public_name my-ppx)
  (kind ppx_rewriter))

From this, Dune creates a "driver": a single executable that can contain multiple ppx transformations and is optimised to run across your codebase.

When Dune finds an executable and runs the build, it generates an executable called my_program.exe in the _build/default/src/ folder.

(executable
  (name my_program))

If you add a public name:

(executable
  (name my_program)
  (public_name my-program))

The executable becomes available at _build/install/default/bin/my-program. This makes it accessible within your project and to users who install your package, similar to adding an install stanza to your dune file.

These are the steps:

I would keep the executable inside the tests folder, but you can put it anywhere you want.

(executable
  ; name of your executable _build/default/.../standalone.exe
  (name standalone)
 
 ; tell dune to only use "standalone.ml" to generate the executable
 ; this isn't strictly necessary, but it's a good practice to keep only the modules
 ; that are part of the target and avoid "Multiple rules generated for ..."
 (modules standalone)
 (libraries ppxlib your_ppx))

let () = Ppxlib.Driver.standalone ()

This exposes your ppx transformations through ppxlib's Driver.standalone, which defines a CLI for running your ppx transformations directly. It also simulates what Dune exposes to your users when your ppx is part of a build step in a project.

My previous post about cram tests explains a few other ways to make the executable available: make your executable available under cram tests. This is the simplest one.

(cram
 ; Tells dune to depend on the standalone exe,
 ; so it will be part of the cram target
 (deps standalone.exe))

Let's create a simple test file called first_step.t and run it with dune runtest to ensure it's working.

  $ ls
  first_step.t
  standalone.exe

I usually keep a Makefile with the commands test, test-watch, and test-promote so I can type the bare minimum to get the job done, as in this Makefile.

Say we have a ppx that performs the smallest possible transformation, a foobar-kind example: it changes the extension [%yay] into the string "Hello future compiler engineer!".

A cram test for it might look like this:

Creates an file with some OCaml code using bash heredoc
https://linuxize.com/post/bash-heredoc
  $ cat > input.ml <<EOF
  > let () = print_endline [%yay]
  > EOF
 
Run the executable from input.ml and print the output to stdout
  $ ./standalone.exe --impl input.ml
  let () = print_endline "Hello future compiler engineer!"

A real example is browser_ppx, a ppx that handles browser-specific code by stripping it out when the target is not JavaScript. This is very useful when sharing code between native and JavaScript. Its documentation lives here and comes from server-reason-react.

A cram test for it might look like this:

  $ cat > input.ml << EOF
  > let%browser_only pstr_value_binding = Webapi.Dom.getElementById "foo"
  > EOF
 
Output goes into output.ml
  $ ./standalone.exe -impl input.ml -o output.ml
 
Format the output with ocamlformat
  $ ocamlformat --enable-outside-detected-project --impl output.ml
  let pstr_value_binding =
    [%ocaml.error
      "[browser_ppx] browser_only works on function definitions. For other \
       cases, use switch%platform or feel free to open an issue in \
       https://github.com/ml-in-barcelona/server-reason-react."]
 
  let make () =
    let pstr_value_binding_2 =
      [%ocaml.error
        "[browser_ppx] browser_only works on function definitions. For other \
         cases, use switch%platform or feel free to open an issue in \
         https://github.com/ml-in-barcelona/server-reason-react."]
    in
    ()
 
Run the compiler
  $ ocamlc -c output.ml
  File "output.ml", line 2, characters 4-15:
  2 |   [%ocaml.error
          ^^^^^^^^^^^
  Error: [browser_ppx] browser_only works on function definitions. For other
         cases, use switch%platform or feel free to open an issue in
         https://github.com/ml-in-barcelona/server-reason-react.
  [2]

The test verifies that the ppx transforms our browser-specific code into an [%ocaml.error ...] and raises the error defined in the ppx. It then runs the compiler and checks the output. The ocamlc command returns error code 2, shown in the last line of the output as [2].

In a real-world scenario like reason-react-ppx, we want to ensure that the ppx transformations preserve the AST location of the original code. The generated code should have the same location as the original code so the editor can highlight the correct line, compiler errors can report the correct line and column, and so on.

For these tests, we use ocamlmerlin to inspect the AST and ensure the location is correct. Here is part of the test:

 
Test some locations in reason-react components
 
Create a dune project file, with "using melange"
  $ cat > dune-project <<EOF
  > (lang dune 3.8)
  > (using melange 0.1)
  > EOF
 
Create a dune file, with a melange.emit stanza
  $ cat > dune <<EOF
  > (melange.emit
  >  (alias foo)
  >  (target foo)
  >  (libraries reason-react)
  >  (preprocess
  >   (pps melange.ppx reason-react-ppx)))
  > EOF
 
Run the build (this runs the reason-react-ppx)
  $ dune build
 
Let's test hovering over parts of the component
 
key={author.Author.name}
_^
 
Use ocamlmerlin to query the type of the expression at the cursor
  $ ocamlmerlin single type-enclosing -position 10:7 -verbosity 0 \
  > -filename component.re < component.re | jq '.value[0]'
  {
    "start": {
      "line": 10,
      "col": 2
    },
    "end": {
      "line": 10,
      "col": 85
    },
    "type": "string",
    "tail": "no"
  }

The key lesson is to use all the tools at your disposal, such as ocamlmerlin, to ensure your ppx works as expected. This is especially useful because many OCaml tools are CLIs:

• ocamlmerlin is an LSP library used by many editors to provide code analysis
• ocamlc is the bytecode OCaml compiler
• ocamlopt is the native OCaml compiler
• ocamldep is the dependency resolver
• ocamldoc is the OCaml documentation generator
• ocamlformat is the OCaml code formatter

For more tools, check the OCaml documentation.

If you're building a ppx, I highly recommend trying cram tests. They might make your development process a bit more enjoyable. They also give you an easy way to add all the cases you want to keep supporting in the future or ensure that language edge cases continue to work.

Check the ppxlib-simple-example repository for a complete example.

Happy testing folks!