query-json is a faster and simpler re-implementation of jq's language in Reason, compiled to a native binary and a JavaScript library.

It's a CLI for running small programs against JSON files, the same idea as sed for text. As a web engineer, I find it essential when debugging HTTP APIs or exploring big JSON files, such as those from AWS Config.

I started the project to create something useful and learn along the way. I especially wanted to learn how to write a parser and a compiler with the OCaml stack, using menhir and sedlex, and then try compiling it to JavaScript.

I'll explain the project, how I made it, the decisions I followed, and some reflections.

I had a vague idea of the theory but no practical experience. The timing was right because I had made styled-ppx, a ppx (PreProcessor Extension) that allows CSS-in-Reason/OCaml. It needs to parse CSS and generate some code for bs-emotion.

I asked @EduardoRFS for help writing a CSS Parser that supports the entire CSS3 specification, and he came up with something. That "something" is a project I want to understand, improve, and maintain over time.

query-json ".store.books | filter(.price > 10)" stores.json

This reads stores.json and runs the query ".store.books | filter(.price > 10)" against it.

The query describes a jq program. It accesses the "store" field and then the "books" field. Because "books" is an array, it filters each item by its "price", keeping prices larger than 10, and prints the resulting list.

[
  {
    "title": "War and Peace",
    "author": "Leo Tolstoy",
    "price": 12.0
  },
  {
    "title": "Lolita",
    "author": "Vladimir Nabokov",
    "price": 13.0
  }
]

A jq program consists of piped operations. Each operation's output becomes the next operation's input, with the JSON itself as the first input. Some pseudo-code to illustrate:

{ /* json */ } | filter | transform | count | .field

To transform the query into operations that run against JSON, we'll divide the problem into 3 steps: parse, compile and run.

Parsing transforms a string into an AST (Abstract Syntax Tree), a data structure with the same information as the input in a shape that's easier to work with. The parser can also return errors if the input is malformed and doesn't follow the rules.

One of the beauties of jq is that all expressions are piped by default, so .store | .books is equivalent to .store.books. I designed the AST so its structure represents that pipe. If you want to know more about jq's language, check its wiki.

For example, when the parser receives .store.books, it returns:

Pipe(Key("store"), Key("books"));

The parser transforms all operations into constructors such as Pipe and Key. These constructors are called Variants.

Variants model values that may assume one of many known variations. This feature resembles enums in other languages, but each variant may optionally carry data. Variants belong to a large group of types called ADTs.

The entire query-json AST is one big recursive variant.

A more complex example parses .store.books | filter(.price > 10):

Pipe(
  Pipe(Key("store"), Key("books")),
  Filter(Pipe(Key("price"), Literal(Number(10))))
);

Here, Pipe represents both the pipe | and .store.books. The parsing tests contain more examples.

The compilation step receives the AST expression and transforms it into code. The compiler is one big recursive pattern match, another great feature of Reason/OCaml. It looks something like this:

let rec compile = (expression, json) => {
  switch (expression) {
  | Empty => empty
  | Keys => keys(json)
  | Key(key, opt) => member(key, opt, json)
  | Index(idx) => index(idx, json)
  | Head => head(json)
  | Tail => tail(json)
  | Length => length(json)
  /* [...] */
}

The left side defines every possible Variant, while the right side defines each operation. These operations transform the JSON. This is where map, filter, reduce, index, etc. are implemented. In the real implementation, many branches call compile recursively.

This is the easier part. The compile step returns a curried function that expects a json as its only argument. We apply the function to this JSON and print the result.

This example describes only the happy path. In reality, the parsing and compilation steps return a result type that allows error handling.

let compile: Ast.expression -> Json.t -> (Json.t list, string) result

We've covered how it works internally and the basic architecture. Developers can use it on their machines in the following ways.

query-json is built with dune, which supports ReasonML out of the box. By default, dune can run the OCaml compiler with different backends. It can compile to bytecode, which runs with the bytecode interpreter, or to binary for a native executable.

All build steps and tests run in our CI on GitHub Actions using Mac, Windows, and Linux images. I distribute pre-built binaries for all architectures through GitHub Releases and the npm registry.

Users can download it directly through npm or from the GitHub release page.

Besides compiling to an executable, query-json also compiles to JavaScript.

Compiling to JavaScript is the part of this blog post I'm most proud of. It didn't take much effort because the tools I used are already mature. Still, releasing 2 distributables from one codebase feels like magic, especially with all of query-json's dependencies: menhir, sedlex, and yojson.

I used js_of_ocaml, or jsoo for short. jsoo is a compiler that takes an intermediate representation from the OCaml compiler, the bytecode I mentioned earlier, and transforms it into JavaScript.

Because dune supports jsoo out of the box, I only needed to add (modes js) to its stanza:

(executable
 (name Js)
 (modes js)
 (libraries console.lib source yojson js_of_ocaml))

After running dune build, I had a big file named Js.bc.js with all the code bundled. That was kind of amazing, tbh.

After compiling to JavaScript, I built a web playground where people could try query-json without installing anything. This made the tool immediately accessible. They could open a browser and start typing, with no downloads or setup.

The playground also brought some nice benefits. I could deploy preview versions for every pull request, users could share bug reports through a single URL, and everything ran faster because it executed locally in the browser.

The playground is built with jsoo and a few cool dependencies: jsoo-react and jsoo-css. You can try it yourself here:

https://query-json.netlify.app

The playground runs query-json on each keystroke, so it also works offline. The official jq playground needs to communicate with a backend, run jq there, and return the response. The difference is night and day.

A playground built as a serverless frontend app is a massive improvement over one that depends on a backend. It's faster, safer, more scalable, and accessible to everybody.

I found jsoo to be a powerful way to run OCaml code in a browser without much hassle. That was a key takeaway for me, and it helped make this project possible. In my opinion, it also offers these benefits beyond distribution:

• Portability: moving code from server to client or vice versa, sharing marshal/unmarshal code, easier contract testing.
• Familiarity: writing the same patterns helps newcomers learn fewer platform-specific rules.
• Usage of OCaml's ecosystem: access to many libraries and ppxs and the latest OCaml features.
• New possibilities: some apps might benefit from server-side rendering, others from moving functionality offline, and many app-specific designs are unblocked by this.

Most of the REPLs for Reason, OCaml, Flow, and ReScript, all written in OCaml, use js_of_ocaml for their playgrounds.

query-json is still young. It supports most of jq's core features, but it has room to grow and perhaps diverge.

I want query-json to support more constructors from the jq language and make operations on JSON easier to run through better error messages and performance.

For me, jq is a double-edged sword: very powerful but also confusing. The number of questions on StackOverflow.com proves that the language has many problems without a solution. If query-json gets a lot of traction, I would diverge from jq's syntax and try to solve those confusing parts.

Another mission of query-json is to push performance forward. We are now implementing most of the missing functionality. Next, we'll explore performance optimizations such as:

• Improving JSON parsing through JSON streaming, or better yet, parsing only the parts that the query needs
• Refactoring it with OCaml multicore (once it's published!)
• Replacing menhir with a hand-written parser

I hope you enjoyed the project and its story. Let me know if these topics interest you; I'm always happy to chat.

Thanks to everyone who reviewed this blog post: Javi, Enric, and Gerard.