Conversation
🦋 Changeset detectedLatest commit: 43aaf61 The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Follows livekit/protocol#1799 renaming the proto field.
u9g
changed the title
Follows livekit/protocol#1799 renaming Job.attempt.
5c16cbf to
0ef9b88
Compare
A run today holds exactly one job per scenario, so a single flaky attempt is indistinguishable from a broken scenario. `Create.Request.repeats` asks for k attempts of every scenario in one run; each Job carries the `scenario_id` and 1-based `attempt` it belongs to so jobs group back into scenarios. `RunMetrics.pass_at_k` (at least one attempt passed) and `pass_pow_k` (every attempt passed) summarize the run over scenarios.
66effb8 to
be5b074
Compare
Follows livekit/protocol#1799 renaming the proto field.
Follows livekit/protocol#1799 renaming Job.attempt.
Follows livekit/protocol#1799 renaming the proto field.
Follows livekit/protocol#1799 renaming Job.attempt.
|
Added The semantics are narrower than "pass^k" suggested, and worth stating plainly because the field comment fixes them for every consumer: A scenario stops being all-or-nothing. The rate is the share of one scenario's samples that must pass for that scenario to pass — The rounding is
|
…rate A scenario is all-or-nothing today, so repeating it can only ever make it harder to pass. min_sample_pass_rate is the share of a scenario's samples that must pass for that scenario to pass, set on Create.Request and mirrored on SimulationRun so the dashboard can render a verdict without being told the bar out of band. 0/unset requires every sample, which is the current behaviour. The run still fails if any scenario fails, and pass@k / pass^k stay informational.
6e3a18d to
c136f73
Compare
| // pass, 0-1; 0/unset requires every sample. A scenario needs | ||
| // round(rate × samples) of them. The run fails if any scenario fails; | ||
| // pass@k and pass^k report over scenarios and neither gates. | ||
| double min_sample_pass_rate = 15; |
There was a problem hiding this comment.
maybe just pass_rate?
| double min_sample_pass_rate = 15; | |
| double pass_rate = 15; |
There was a problem hiding this comment.
Done, on both the request and the mirrored field on SimulationRun.
0 samples means run nothing and a 0 pass rate means require nothing, so neither zero value can stand for "the caller said nothing" without a remap that lives only in a comment. Absence carries that instead, and an explicit 0 becomes a rejectable input. SimulationRun.samples stays plain: a materialized run always has a real count, 1 when it ran once.
pass_rate is meaningless without samples > 1, so the two are one decision and a request carrying only one of them is not a state worth representing. Inside Sampling both are plain: 0 samples and a 0 pass rate are naturally invalid, so neither needs a presence bit to stand in for "the caller said nothing".
7d8e3dd to
cb4733f
Compare
2 participants
A run today holds exactly one job per scenario, so a single flaky attempt is indistinguishable from a broken scenario. Repeating a scenario only helps if repeating can also change its verdict, which is what the pass rate is for.
SimulationRun.Sampling:samples(attempts per scenario, > 1) andpass_rate(the share of a scenario's samples that must pass for that scenario to pass, 0-1). A scenario needsround(rate × samples)passing samples, so0.34of 3 needs 1 and0.67of 3 needs 2;1requires every sample. Both are required when the message is present.SimulationRun.Create.Request.sampling, mirrored onSimulationRun.sampling. Absent runs each scenario once and requires it to pass, which is today's behaviour. Requiresscenario_group. The mirror lets the dashboard render a scenario's verdict without being told the bar out of band.SimulationRun.Job.scenario_id/sample: every attempt of one scenario shares the id, so a run's jobs group back into scenarios. New field numbers; 9 stays reserved.A scenario stops being all-or-nothing:
2/3can now be a pass. The run still fails if any scenario fails, so there is no run-level percentage. pass@k and pass^k remain reporting metrics over scenarios and neither gates.pass_rateis meaningless withoutsamples > 1— at one sampleround(rate × 1)is either a no-op or always-passes — so the two travel together rather than as two independently-optional fields that can be combined into a request that means nothing. InsideSamplingboth are plain scalars, because0 samplesand a0pass rate are naturally invalid and so already say "unset" without a presence bit.Note for consumers:
Sampling.pass_rateis the threshold a run was given, andRunMetrics.scenario_pass_rateis the observed share of passing jobs. They are a bar and a measurement, not two readings of the same thing.Consumer migration.
samplesandpass_ratewere briefly flat fields on this branch, never on main and never released, sosamplingreuses their field numbers rather than reserving them. livekit/agents-private#457 is merged and pins a commit of this branch ingo.mod, so it needsGetSampling().GetSamples()and a repin. livekit/livekit-cli#975 (lk agent simulate --samples/--pass-rate) and livekit/web#6190 follow.