expbook

A local, serverless experiment tracker: one run is one directory of JSON.

Every training script prints its own numbers and then they are gone, or they live in a notebook cell nobody reruns, or in a spreadsheet somebody forgot to update. Comparing "did this run of the model actually beat the last one" ends up done from memory. expbook is the smallest thing that fixes that: wrap a run in track() and it writes its params, its metric history, and its final summary straight to disk as plain JSON. There is no daemon to start, no server to point a browser at, and no database file that can get corrupted mid-run.

Install

$ pip install git+https://github.com/jmweb-org/expbook
$ uv add git+https://github.com/jmweb-org/expbook   # if you use uv

Not on PyPI. Only dependencies are typer and rich, for the CLI.

Quickstart

from expbook import track

with track("baseline", params={"lr": 0.01, "epochs": 20}) as run:
    for step in range(20):
        loss = train_one_epoch(step)
        run.log(step=step, loss=loss)
    run.metric("final_auc", 0.87)
    run.artifact("model.pkl")

$ expbook list baseline
exp       run       status     duration  metrics
baseline  a1b2c3d4  completed  4.2s      final_auc=0.87

This creates runs/baseline/<timestamp>-<id>/ holding params.json, metrics.jsonl (one line per run.log() call), summary.json (final metrics, duration, git commit, Python and platform versions), and an artifacts/ folder for anything copied in with run.artifact().

Usage

Tracking a run

with track("exp-name", params={...}) as run:
    run.log(step=1, loss=0.5)        # appended to metrics.jsonl
    run.metric("final_auc", 0.87)    # goes into summary.json
    run.artifact("model.pkl")        # copied into artifacts/

• track(exp, params=None, root="runs", run_id=None) starts a run. params.json is written the moment the run starts, before anything else happens, so a run that crashes early still leaves a record of what it was trying.
• run.log(step=None, **metrics) appends one JSON object per call. Nothing is buffered in memory, so a training loop that gets killed keeps every step it already logged.
• run.metric(name, value) sets (or overwrites) a final metric, recorded in summary.json once the run ends.
• run.artifact(path) copies a file or a directory into the run's artifacts/ folder and returns the destination path.

If the block raises, the summary records status: "failed" with the full traceback, and then the original exception is re-raised — track() records a failure, it does not hide one:

with track("exp-name") as run:
    run.log(step=0, loss=float("nan"))
    raise ValueError("loss diverged")
# summary.json: {"status": "failed", "traceback": "...ValueError: loss diverged\n"}
# and the ValueError still propagates to the caller

CLI

$ expbook list [exp]                        # table of runs, newest first
$ expbook show <run>                        # params, metrics, status, environment
$ expbook compare <runA> <runB>              # params + metrics side by side, with deltas
$ expbook best <exp> --metric auc            # highest auc in the experiment
$ expbook best <exp> --metric loss --min     # lowest loss in the experiment

Every command takes --json for scripting and --root to point at a tracking directory other than ./runs. A <run> reference can be the path to the run directory itself, the <experiment>/<run-id> shorthand, an id prefix (<experiment>/a1b2), or a bare id searched across every experiment.

Demo

A tiny numpy linear regression, tracked across three runs at different learning rates:

import numpy as np
from expbook import track

rng = np.random.default_rng(0)
x = rng.normal(size=200)
y = 3.0 * x + 1.0 + rng.normal(scale=0.3, size=200)

for lr in (0.01, 0.1, 0.5):
    with track("linreg", params={"lr": lr, "steps": 200}) as run:
        w, b = 0.0, 0.0
        for step in range(200):
            pred = w * x + b
            err = pred - y
            grad_w = (2 * err * x).mean()
            grad_b = (2 * err).mean()
            w -= lr * grad_w
            b -= lr * grad_b
            if step % 40 == 0:
                run.log(step=step, mse=float((err**2).mean()))
        mse = float(((w * x + b - y) ** 2).mean())
        run.metric("mse", mse)
        run.metric("w", w)
        run.metric("b", b)

Real output from that script, run against this checkout:

$ expbook list linreg
exp     run     status     duration  metrics
linreg  lr-05   completed      0.0s  b=0.9738, mse=0.09421, w=2.979
linreg  lr-01   completed      0.0s  b=0.9738, mse=0.09421, w=2.979
linreg  lr-001  completed      0.0s  b=0.9604, mse=0.09898, w=2.909

$ expbook compare linreg/lr-001 linreg/lr-05
params
key    linreg/20260712T121852Z-lr-001  linreg/20260712T121852Z-lr-05
lr     0.01                            0.5
steps  200                             200
metrics
key  linreg/20260712T121852Z-lr-001  linreg/20260712T121852Z-lr-05  delta
b    0.9604                          0.9738                         +0.01334
mse  0.09898                         0.09421                        -0.004771
w    2.909                           2.979                          +0.07028

$ expbook best linreg --metric mse --min
run           linreg/20260712T121852Z-lr-05
status        completed
duration      0.0s
started       2026-07-12T12:18:52.687240+00:00
ended         2026-07-12T12:18:52.695087+00:00
git sha       9b35426b2b953a7a0067fc9b249ebe3e6340f5b4
python        3.12.3
platform      Linux-6.17.0-35-generic-x86_64-with-glibc2.39
steps logged  5
artifacts     -
params
key    value
lr     0.5
steps  200
metrics
key  value
b    0.9738
mse  0.09421
w    2.979

lr=0.5 overshoots less than it looks like it should here only because the loss surface is a simple convex bowl; on anything less friendly a learning rate that large is usually the first thing to blow up a run, which is exactly the kind of thing worth being able to see across runs instead of just in the last terminal that happened to be open.

Where this fits with the rest of the suite

expbook, pytest-metricguard and evalgate answer three different questions about a metric, and none of them do the others' job.

expbook is the history: it records what every run's params and metrics were, so "what did we try last week" has an answer that is not "check the terminal scrollback." It has no opinion about whether any given number is good or bad.

pytest-metricguard is the CI gate: it fails a test when a metric it is watching drops past a fixed tolerance from a committed baseline. It does not care what the run's other params were, only whether this one number moved too much.

evalgate is the judgment call for a metric drop that pytest-metricguard flagged, or one found by comparing two expbook runs by hand: it runs a significance test to tell a real regression apart from sampling noise on a small eval set, so a CI gate does not flap on a couple of examples flipping.

In practice: log every run with expbook so there is something to compare. Gate CI on the metrics that matter with pytest-metricguard. When a gate fails or a comparison looks close, run evalgate before treating the drop as real.

What it does not do

No database, no web UI, no run comparison across machines beyond copying the runs/ directory around, and no locking — two processes writing to the same run directory at once will interleave metrics.jsonl lines rather than corrupt them, but nothing stops it happening. It is built for one person's local runs, or a runs/ directory synced or committed deliberately, not for a team dashboard.

License

MIT. See LICENSE.