Skip to content

Goalpost: v19 bounded-memory and CPU performance contract #758

Description

@flyingrobots

Objective

Make performance a versioned, reviewable v19 contract rather than an informal local observation.

git-warp must prove both:

  1. causal materialization and observation do not regress materially in CPU cost; and
  2. a graph whose logical size is substantially larger than the configured JavaScript heap can still be observed to completion with a bounded working set.

This goalpost extends #734 and #738. It is not complete merely because one optimized path is faster once.

Required benchmark surfaces

Execution modes

  • cold retained materialization from Git history;
  • warm reuse from git-cas materialization state;
  • incremental materialization after a bounded causal suffix;
  • streamed observation, including time to first Reading and completion;
  • cache-miss and cache-hit behavior reported separately.

Metrics

  • process user CPU, system CPU, and total CPU;
  • wall-clock duration as a diagnostic, not the only gate;
  • time to first Reading and steady-state throughput;
  • peak RSS and peak JavaScript heap;
  • logical corpus size, tick count, node/property/edge counts, and emitted Reading count;
  • Git command/object read/object write counts where instrumentation permits;
  • git-cas hit, miss, staged, promoted, and restored-byte evidence.

Oversized-graph proof

The corpus must be generated and consumed lazily. The benchmark subprocess must run with an explicit --max-old-space-size and operate on a logical graph at least several times larger than that cap. The proof fails if it silently creates a complete fixture array, materializes the full graph, omits expected Readings, exceeds the memory budget, or relies on an unbounded compatibility API.

Merge gates

  • correctness, expected cardinality, termination, and heap-cap compliance are hard deterministic gates;
  • base and head execute on the same runner and toolchain;
  • CPU comparisons use warmups plus repeated samples, medians, an absolute noise floor, and a reviewed relative-regression budget;
  • benchmark ordering must avoid systematically favoring base or head;
  • the initial calibration change may establish the baseline, but subsequent baseline movement requires explicit review and rationale;
  • shared-runner wall-clock noise must never be disguised as a precise CPU contract.

Publication

Every main-branch run publishes a machine-readable, versioned benchmark result and a human-readable summary. Pull requests show base/head deltas. Historical results are exposed without force-pushing generated Git branches; GitHub Pages artifacts or an equivalent append-safe publication path are preferred.

Acceptance criteria

  • A versioned corpus and result schema are committed.
  • Cold, warm, incremental, and streamed scenarios have reproducible runners.
  • The oversized-graph subprocess completes under its explicit heap cap.
  • Peak RSS/heap evidence and expected Reading cardinality are captured.
  • Base/head CPU comparison is calibrated and blocks material regressions.
  • Main publishes durable JSON and human-readable benchmark summaries.
  • Pull requests expose deltas and fail closed when required evidence is absent.
  • Baseline updates require an intentional reviewed change.
  • The v19 release witness cites benchmark runs and the active thresholds.

Non-goals

  • Claiming host-independent nanosecond precision.
  • Treating a microbenchmark as end-to-end proof.
  • Optimizing benchmark-only code paths.
  • Allowing performance thresholds to weaken correctness or causal evidence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:testingPrimary work area: testing.priority:asapImmediate release pressure.status:activeSomeone is actively working this issue.type:goalpostRoadmap umbrella milestone issue.

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions