Skip to content

A results.json the report cannot read renders as "nothing measured yet", at exit 0 #45

Description

@vyncint

TodayResults::load (crates/launchbound-bench/src/run.rs:367-370) is read_to_string(path).ok()? followed by serde_json::from_str(&text).ok(), and RunDir::load (crates/launchbound-report/src/build.rs:35) takes its None to mean the measurement box has not run. So a missing results.json, a truncated one, an empty one, null, [], a document with a different schema, and a directory called results.json all render the same report. Measured against 2.0.0 on the run directory this repository ships, runs/reduce-stable-metal, with its results.json replaced:

results.json = '{'                          exit=0   launchbound report — reduce-stable · gate cc metal · unmeasured · no device — nothing measured yet
results.json = ''                           exit=0   (identical)
results.json = 'null'                       exit=0   (identical)
results.json = '[]'                         exit=0   (identical)
results.json = '{"schema":"results.v1"}'    exit=0   (identical)
results.json = <a directory>                exit=0   (identical)
stderr: empty, in every case

$ launchbound report badrun --json | jq -r .measurement_kind
unmeasured                                  # and the document validates against schemas/report.v1.json, 0 errors

With the shipped file intact the same command prints CHOSEN: c1-43e865c2e1620697 … measured.

model --results goes through the same loader, so it cannot say why either (crates/launchbound-cli/src/main.rs:602-603):

$ launchbound model reduce-flip --cc 8.6 --results empty.json
error: cannot read results.v1 at "empty.json"
$ launchbound model reduce-flip --cc 8.6 --results shape.json        # {"schema":"results.v1"}
error: cannot read results.v1 at "shape.json"
$ launchbound model reduce-flip --cc 8.6 --results /nonexistent.json
error: cannot read results.v1 at "/nonexistent.json"

Why it is worth fixing — the run directory is the hand-off between two machines. stage writes the plan here, the measurement box writes results.json, and the file comes back by whatever copies it. A partial copy, the wrong file, or a results.v2 from a newer runner all read as "the box has not run yet": exit 0, nothing on stderr, and a JSON report that is schema-valid and says unmeasured. The two conditions call for opposite actions — wait, or go and look — and nothing tells them apart. tune renders this same report at the end of a run (main.rs:533), so the last thing a run prints can be wrong in the same way.

The checkpoint writer is atomic — temp file, then rename (run.rs:373-381) — so a crash on the machine that measures leaves results.json.tmp, not a torn file. The exposure is everything after the box, which is the part no test on either machine sees. verdicts.json already gets this right two lines above: its schema tag is checked by name and a mismatch is unsupported verdicts schema …, exit 2 (build.rs:28-33). results.json deserves the same.

Fix — make Results::load return Result<Option<Results>, String>: NotFound is Ok(None); every other I/O error and every parse error is Err with the path and serde's message. report exits 2 on Err with the reason on stderr and no report on stdout; model --results prints the cause instead of "cannot read". Check the schema tag against results.v1 the way verdicts.v1 is checked, so a future results.v2 is refused by name rather than read as nothing.

Done when — a truncated, empty, wrong-schema or directory results.json makes report exit 2 with the cause on stderr and nothing on stdout; a missing one still renders "nothing measured yet"; model --results names the cause; a test in launchbound-report covers each shape; tune's end-of-run report inherits the behaviour.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions