Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 16 additions & 12 deletions cli/cmd/mcp_caveats.go
Original file line number Diff line number Diff line change
Expand Up @@ -71,20 +71,24 @@ var mcpCaveatsCore = []mcpCaveat{
},
{
ID: mcpCaveatReaperDeadline,
Text: "A run still RUNNING past its deadline is marked FAILED by a reaper that checks every 60 seconds. The " +
"deadline counts from the run's creation, so time spent waiting to start counts. A current Kates API sets " +
"it when it creates the run: how long the run is set to last, at most kates.engine.max-duration-ms " +
"(default 7200000), plus kates.engine.reaper-grace-ms (default 300000). That length is the spec's " +
"durationMs, twice that for INTEGRITY, which reads its records back for up to as long again, or a " +
"scenario's phases added up; an INTEGRATION_CDC run, which no duration bounds, gets the cap. That API " +
"refuses with 400 a request set to last longer than the cap, and when it fails a run, each task that had " +
"not finished gets an error that starts \"Timeout:\" and the others keep their results. An older Kates " +
"API fails every run 30 minutes after its creation (max-duration-ms, default 1800000), whatever the run " +
"was set to last, and deletes the task results of each run it fails, and of each run a restart left " +
"RUNNING, so such a run has no tasks. Neither /api/health nor a run says which of the two the API is.",
Text: "A run still PENDING, RUNNING or STOPPING past its deadline is marked FAILED by a reaper that checks " +
"every 60 seconds. The deadline counts from the run's creation, so time spent waiting to start counts. A " +
"current Kates API sets it when it creates the run: how long the run is set to last, at most " +
"kates.engine.max-duration-ms (default 7200000), plus kates.engine.reaper-grace-ms (default 300000). That " +
"length is the spec's durationMs, twice that for INTEGRITY, which reads its records back for up to as " +
"long again, or a scenario's phases added up; an INTEGRATION_CDC run, which no duration bounds, gets the " +
"cap. That API refuses with 400 a request set to last longer than the cap, and when it fails a run, each " +
"task that had not finished gets an error that starts \"Timeout:\" and the others keep their results. A run " +
"with no task, as a PENDING run whose submission never stored it RUNNING has none, gets one, " +
"<id>-submission, with that error. Kates API 1.25.0 holds only RUNNING runs to the deadline, and leaves a " +
"PENDING or STOPPING run as it is. An older Kates API fails every run 30 minutes after its creation " +
"(max-duration-ms, default 1800000), whatever the run was set to last, and deletes the task results of " +
"each run it fails, and of each run a restart left RUNNING, so such a run has no tasks. Neither " +
"/api/health nor a run says which of these the API is.",
Refs: []string{
"kates/src/main/resources/application.properties:336-350",
mcpJava + "engine/TestTimeoutReaper.java:39-47,50-136",
mcpAnchoredRef(mcpJava+"engine/TestTimeoutReaper.java:55-56,64-73,75-205",
"TestResult.TaskStatus.STOPPING", "kates.engine.reaper-grace-ms", "\"-submission\"", "\"Timeout: still \""),
mcpJava + "engine/TestOrchestrator.java:132-154,186-189,354,1109-1168,1234-1243,1298-1330",
mcpJava + "persistence/EntityMapper.java:158-167",
"kates/src/main/resources/db/migration/V24__run_planned_duration.sql:1-13",
Expand Down
2 changes: 1 addition & 1 deletion cli/cmd/mcp_tools_runs.go
Original file line number Diff line number Diff line change
Expand Up @@ -2361,7 +2361,7 @@ var mcpCaveatsRuns = []mcpCaveat{
Refs: []string{
mcpAnchoredRef(mcpJava+"service/RunRetention.java:82",
`auditService.record("DELETE", "test", id, auditDetails, actor);`),
mcpAnchoredRef(mcpJava+"service/TestCleanupScheduler.java:99", "Actor.SCHEDULER"),
mcpAnchoredRef(mcpJava+"service/TestCleanupScheduler.java:97", "Actor.SCHEDULER"),
mcpAnchoredRef(mcpJava+"audit/AuditResponseFilter.java:42-62",
"public void filter(ContainerRequestContext request", "if (actor == null)", "static String idOf("),
mcpAnchoredRef(mcpJava+"security/GrpcApiKeyInterceptor.java:67-70,100,156-159",
Expand Down
2 changes: 1 addition & 1 deletion docs/book/02-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -397,7 +397,7 @@ The events go to `kates-test-events`, a topic the Kates API creates for itself o

Every replica of the Kates API reads `kates-test-events` in one [consumer group](appendix-a-glossary.md#gl-consumer-group), `kates-webhooks`, so one replica handles each event, and a replica that starts resumes from the group's [committed offset](appendix-a-glossary.md#gl-committed-offset). A group with no committed offset, as on its first start, reads from the oldest event the topic holds. The `processed_events` table remembers the events handled in the last 7 days (`kates.outbox.processed-events-retention-days`), so an event read twice fires its webhooks once, and the Kates API skips any event older than that.

What lives in the Kates API's memory goes with its pod. When the Kates API shuts down, it stops the runs in flight and stores them as `FAILED` with the error `Server shutdown`. When the pod dies without shutting down, those runs stay `RUNNING` until the Kates API starts again, and it marks them `FAILED` as it starts, each task that had not finished with the error `Recovered: test was orphaned after server restart`. Either way, each task keeps the figures it last recorded, and every heatmap and every built report is gone. `kates clean` goes further: it deletes the namespace that holds the bundled PostgreSQL, and every stored run with it.
What lives in the Kates API's memory goes with its pod. When the Kates API shuts down, it stops the runs in flight and stores them as `FAILED` with the error `Server shutdown`. When the pod dies without shutting down, those runs stay `RUNNING` until the Kates API starts again, and it marks them `FAILED` as it starts, each task that had not finished with the error `Recovered: test was orphaned after server restart`. In both cases, a run whose tasks were still being started stays `PENDING` until five minutes after the time it was set to last, when the Kates API fails it with an error that starts `Timeout: still pending`. Either way, each task keeps the figures it last recorded, and every heatmap and every built report is gone. `kates clean` goes further: it deletes the namespace that holds the bundled PostgreSQL, and every stored run with it.

## Data Model

Expand Down
4 changes: 3 additions & 1 deletion docs/book/11-api-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -290,7 +290,9 @@ The `spec` in the response is the merged one: the request's values, and the LOAD

Run IDs are 8-character UUID prefixes. `status` moves through `PENDING` and `RUNNING`, and ends at `DONE` or `FAILED`. There is no cancelled status: `POST /api/tests/{id}/cancel` stores the run as `FAILED` and answers `{"id": ..., "status": "FAILED", "reason": "cancelled", ...}`, and each task it stopped carries the error `Cancelled by user`. The cancel also ends the run's workers and gives back its place among the `kates.engine.max-concurrent-tests` running tests. A run that finishes on its own while the cancel is being made keeps its own ending, and the cancel answers `409`.

A run still `RUNNING` five minutes (`kates.engine.reaper-grace-ms`) after the time it was set to last, counted from its creation, is stopped and stored as `FAILED` too. That time is its `durationMs`, twice that for INTEGRITY, or a scenario's phases added up; INTEGRATION_CDC, which has no duration of its own, gets `kates.engine.max-duration-ms`. Each task that had not finished carries an error that starts `Timeout:`, and the tasks that had keep their results.
A run still `PENDING`, `RUNNING` or `STOPPING` five minutes (`kates.engine.reaper-grace-ms`) after the time it was set to last, counted from its creation, is stopped and stored as `FAILED` too. That time is its `durationMs`, twice that for INTEGRITY, or a scenario's phases added up; INTEGRATION_CDC, which has no duration of its own, gets `kates.engine.max-duration-ms`. Each task that had not finished carries an error that starts `Timeout:`, and the tasks that had keep their results.

A run stays `PENDING` when the Kates API stops while it starts the run's tasks, and a restart leaves it so until that time is up. Such a run has no task, so it gets one, `<id>-submission`, that carries the error. Only a Kates API before 1.25.0 stores a run as `STOPPING`, and a run one left so ends the same way.

#### GET /api/tests

Expand Down
2 changes: 1 addition & 1 deletion docs/book/12-deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ The Kates API keeps every run in PostgreSQL. The `kates` chart runs its own Post

Whichever database holds them, each replica of the Kates API sweeps out old runs when it starts and every 24 hours after. A sweep deletes the `DONE` and `FAILED` runs created more than `kates.cleanup.retention-days` days ago, 90 by default, so a lower setting deletes as soon as a replica starts with it. It deletes each run as `DELETE /api/tests/{id}` does, with its results, oldest first and 1,000 at a time. Each leaves a row in the audit log naming `system:scheduler`: action `DELETE`, type `test`, the run's ID, and the details `retention sweep: created before <cutoff> (kates.cleanup.retention-days=<n>)`. When two replicas find the same run at once, it is deleted, and audited, once. An error ends that sweep with an `ERROR` line in the Kates API's log, and the next sweep goes on from there.

A sweep leaves a run still `RUNNING` alone, however old it is, and the first sweep after the run ends deletes it. The Kates API fails a run still `RUNNING` five minutes after the time it was set to last. A run left `PENDING` by a submission that died, or `STOPPING` by an older version of Kates, never ends, so no sweep deletes it. Delete such a run with `DELETE /api/tests/{id}`, or `kates test delete`.
A sweep leaves a run still `PENDING`, `RUNNING` or `STOPPING` alone, however old it is. The Kates API fails such a run five minutes after the time it was set to last, as [Test Management](11-api-reference.md#test-management) describes, and a sweep deletes it once it is older than the retention.

To keep runs for less time, set `cleanup.enabled: true` in the `kates` chart's values. Its CronJob then deletes the finished runs older than `cleanup.retentionDays`, through [DELETE /api/tests](11-api-reference.md#delete-apitests), with the key from the Secret the `apiKey` values name. The table gives the values and their defaults:

Expand Down
2 changes: 1 addition & 1 deletion docs/book/appendix-a-glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -698,7 +698,7 @@ In this book, one service sharing a Kafka cluster: one `KafkaUser` and one topic

### Test Run {#gl-test-run}

One execution of a test spec, with its own ID; its status is PENDING, RUNNING, STOPPING, DONE or FAILED. The Kates API runs at most three at once by default (`kates.engine.max-concurrent-tests`) and answers another with 429. It refuses a run set to last longer than two hours by default (`kates.engine.max-duration-ms`), and fails a run still RUNNING five minutes after its duration is up. See [REST API Reference](11-api-reference.md#test-management).
One execution of a test spec, with its own ID; its status is PENDING, RUNNING, STOPPING, DONE or FAILED. The Kates API runs at most three at once by default (`kates.engine.max-concurrent-tests`) and answers another with 429. It refuses a run set to last longer than two hours by default (`kates.engine.max-duration-ms`), and fails a run still PENDING, RUNNING or STOPPING five minutes after its duration is up. See [REST API Reference](11-api-reference.md#test-management).

### Test Spec {#gl-test-spec}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -707,9 +707,9 @@ public TestRun refreshStatus(String runId) {
* Best-effort FAILED for a run whose registration blew up half-way.
*
* <p>If the save succeeded and only the event failed, the row is left
* PENDING — a state nothing scans: orphan recovery looks for RUNNING and the
* timeout reaper only reaps RUNNING, so the run would sit there forever
* looking like it was about to start. Failing to write this is not worth
* PENDING, which orphan recovery does not read: the run would look as if
* it were about to start until the timeout reaper failed it, once its
* deadline had passed. Failing to write this is not worth
* masking the original error, so it only logs.
*/
private void markStrandedAsFailed(TestRun run) {
Expand Down Expand Up @@ -2020,11 +2020,11 @@ private void fireEvent(TestRun run, TestLifecycleEvent.EventKind kind) {
*
* <p>It ends the run the way the timeout reaper does, because nothing polls
* or reaps a FAILED run afterwards: {@link #refreshStatus} returns early for
* it and the reaper only scans RUNNING. So {@link #settle} stops its
* workers and hands back the concurrency slot and the per-run meters here,
* before FAILED is written. The write is a compare-and-set on the status
* read, so a run that ends on its own in between keeps its ending; one that
* only moved from PENDING to RUNNING is cancelled on the second pass.
* it and the reaper reads only runs that have not ended. So {@link #settle}
* stops its workers and hands back the concurrency slot and the per-run
* meters here, before FAILED is written. The write is a compare-and-set on
* the status read, so a run that ends on its own in between keeps its
* ending; one that only moved PENDING to RUNNING is cancelled on pass two.
*
* <p>A run whose tasks are still being submitted has no handles yet, so
* there is nothing here to stop. Its submission's own write is conditional
Expand Down
Loading
Loading