Ninja build system generator

This module contains helper dcg predicates to generate ninja build files akin to the ninja_syntax.py python module distributed by ninja. You can use these predicates if you want to generate your own build.ninja build file.

Example usage:

build_graph -->
  rule(cp, "cp $in $out"),
  build(["input.txt"], cp, ["output.txt"]).

main -->
  phrase(build_graph, L),
  open("build.ninja", write, Stream),
  string_codes(S, L),
  write(Stream, S),
  close(Stream).

Then build.ninja contains the following build specification:

rule cp
  command = cp $in $out

build input.txt: cp output.txt

See the ninja build format documentation for generating more complex build files.

Installation

You can install this pack in swi-prolog as follows:

?- pack_install(ninja).
% Contacting server at https://www.swi-prolog.org/pack/query ... ok
Install ninja@0.2 from https://github.com/kwon-young/ninja/archive/v0.2.zip Y/n? 
% Contacting server at https://www.swi-prolog.org/pack/query ... ok
% "v0.2.zip" was downloaded 2 times
Package:                ninja
Title:                  Ninja build system generator
Installed version:      0.2
Author:                 Kwon-Young Choi <kwon-young.[email protected]>
Maintainer:             Kwon-Young Choi <kwon-young.[email protected]>
Packager:               Kwon-Young Choi <kwon-young.[email protected]>
Home page:              https://github.com/kwon-young/ninja
Download URL:           https://github.com/kwon-young/ninja/releases/*.zip
Provides:               ninja
Install "ninja-0.2.zip" (15,854 bytes) Y/n? 
true.

Documentation

You can find a documented list of predicate here.

You can generate the documentation locally, either statically in the subdirectory docs:

$ swipl prolog/ninja.pl
?- doc_save('.', [doc_root('docs'), recursive(true)]).

Or as a server:

?- doc_server(4000).

Requirements

This library only depends on prolog librairies distributed by default and autoloaded by swi-prolog:

• dcg/basics
• dcg/high_order
• option
• error

In order to use this library, you will need to have a good understanding of

• the ninja build system file format
• Prolog Definite Clause Grammar (DCG). Here is a very good primer on the subject.

Motivation

I have always been in search of a good Domain Specific Language (DSL) for build system to do general data processing. This project is my attempt at one.

But first, we need to define what is a build system or a build system generator. Keep in mind that these definitions are highly subjective.

What is a build system ?

A build system is a program which job is to execute other programs by following a build graph. A build graph is often a file that specify the commands to run. Each commands form a node in the graph, for which every input files an output files are clearly identified. By using the same file as output of a node to the input of another node, we can connect the nodes as a graph.

The properties of a good build system are:

• Parallelism: build steps that do not depend on each other should be ran in parallel, so that the whole build finishes as fast as possible
• Caching: build steps should be only rebuilt only if outputs are out of dates

Examples of build systems are:

• make
• ninja
• tup
• redo
• biomake
• and many, many, many, many ... others

Although build systems are very useful, writing the build graph manually can be very tedious. Often, build graph will consist of hundred or thousands (or more) build steps connected in a very complex graph.

This is where build system generator or meta build system come in. Build system generator can translate a build graph specified in a custom DSL or programming language into a build graph that can be executed by make or ninja or another build system. This is what CMake do for instance where its DSL is optimized for program compilation.

Most of the time, build system generators also are build systems. For example, we could consider that make is its own build system generator when using its templating mechanism such as static patterns. In my view, any syntax or DSL that allows you to avoid specifying the full build graph is a build system generator.

Here, ninja is the exception by requiring the full explicit build graph as its input.

General data processing

The traditional use of build system is program compilation. The rules to compile a program are mostly well known and are embodied in well known meta build system such as autotools, CMake, Bazel, etc. Almost every programming language has its own meta build system.

However, it turns out that by following our dichotomy between a build system and a build system generator, we notice that build systems can be used for much more things than just program compilation. Build system can do general data processing but you need to pick your poison on which DSL you will use to specify your build graph.

A very common choice is to use GNU Make syntax with automatic variables, static patterns, etc. However, you will be very quickly limited in how expressive you can be. Here is a use-case I have often encountered that cannot be cleanly expressed with Make.

Let's say, we want to process PDFs. We want to split out each page of each PDF, convert each page as an image, OCR the images and finally concatenate each page text into a single text file. Each PDF can have a different number of pages.

Notice that we have one independent graph per PDF. This can be cleanly expressed using static patterns in Make. However, the split and the join at the pages level have to be specified manually for each PDF, which defeat the use of static patterns anyway.

Using prolog DCG to generate a build graph

Here is how you could use a prolog DCG to specify the above graph.

First, let's specify the knowledge we have of the PDF files we want to process. Namely, a unique stem for each PDF file and the number of pages for each stem.

stem(foo).
stem(bar).

page(foo, 1).
page(foo, 2).
page(bar, 1).
page(bar, 2).
page(bar, 3).

Next, we will write DCG rules that builds the name of all the different files we need for our build graph:

pdf(Stem) -->
  atom(Stem), ".pdf".

pdf_page(Stem, Page) -->
  atom(Stem), "-", number(Page), ".pdf".

all_pages(Goal, Stem) -->
  foreach(page(Stem, Page), call(Goal, Stem, Page), " ").

image(Stem, Page) -->
  atom(Stem), "-", number(Page), ".jpg".

ocr_page(Stem, Page) -->
  atom(Stem), "-", number(Page), ".txt".

ocr(Stem) -->
  atom(Stem), ".txt".

Next, we will describe the various commands to run as ninja rules:

rules -->
  rule(split, "split $in $out"),
  rule(convert, "convert $in $out"),
  rule(ocr, "ocr $in $out"),
  rule(cat, "cat $in $out").

Now, let's write the build graph for a single PDF:

graph(Stem) -->
  build([all_pages(pdf_page, Stem)], split, [pdf(Stem)]),
  foreach(page(Stem, Page), (
    build([image(Stem, Page)], convert, [pdf_page(Stem, Page)]),
    build([ocr_page(Stem, Page)], ocr, [image(Stem, Page)])
  )),
  build([ocr(Stem)], cat, [all_pages(ocr_page, Stem)]).

And finally, the full build graph with rules:

graph -->
  rules,
  foreach(stem(Stem), graph(Stem)).

This DCG will generate the following build.ninja file:

rule split
  command = split $in $out
rule convert
  command = convert $in $out
rule ocr
  command = ocr $in $out
rule cat
  command = cat $in $out
build foo-1.pdf foo-2.pdf: split foo.pdf
build foo-1.jpg: convert foo-1.pdf
build foo-1.txt: ocr foo-1.jpg
build foo-2.jpg: convert foo-2.pdf
build foo-2.txt: ocr foo-2.jpg
build foo.txt: cat foo-1.txt foo-2.txt
build bar-1.pdf bar-2.pdf bar-3.pdf: split bar.pdf
build bar-1.jpg: convert bar-1.pdf
build bar-1.txt: ocr bar-1.jpg
build bar-2.jpg: convert bar-2.pdf
build bar-2.txt: ocr bar-2.jpg
build bar-3.jpg: convert bar-3.pdf
build bar-3.txt: ocr bar-3.jpg
build bar.txt: cat bar-1.txt bar-2.txt bar-3.txt