"Current" recall is not actually current but cumulative so far #108
Labels
bug
Something isn't working
Milestone
Comments
daverigby
added a commit
that referenced
this issue
Jun 14, 2024
Add progress bars for each of the main phases of an experiment -
Setup, Populate and Run. For Populate and Run we require the total
number of records / queries to show a useful end progress so far.
For the Run phase we include the current latency and recall(*) values,
these require additional metrics of how many records have been
upserted so far (note that we generally upsert in batches, so the
existing number of Population requests is not sufficient.
(*) For recall we cannot calculate the current metric (last 10s), as
we only have a single histogram accumulating the results - instead
this shows the overall recall so far. There's an improvement
raised to fix this (#108).
daverigby
added a commit
that referenced
this issue
Jun 17, 2024
Add progress bars for each of the main phases of an experiment -
Setup, Populate and Run. For Populate and Run we require the total
number of records / queries to show a useful end progress so far.
For the Populate phase we include the current rate of upsert. For the
Run phase we include the current latency and recall(*) values, these
require additional metrics of how many records have been upserted so
far (note that we generally upsert in batches, so the existing number
of Population requests is not sufficient.
(*) For recall we cannot calculate the current metric (last 10s), as
we only have a single histogram accumulating the results - instead
this shows the overall recall so far. There's an improvement
raised to fix this (#108).
daverigby
added a commit
that referenced
this issue
Jun 17, 2024
Add progress bars for each of the main phases of an experiment -
Setup, Populate and Run. For Populate and Run we require the total
number of records / queries to show a useful end progress so far.
For the Populate phase we include the current rate of upsert. For the
Run phase we include the current latency and recall(*) values, these
require additional metrics of how many records have been upserted so
far (note that we generally upsert in batches, so the existing number
of Population requests is not sufficient.
(*) For recall we cannot calculate the current metric (last 10s), as
we only have a single histogram accumulating the results - instead
this shows the overall recall so far. There's an improvement
raised to fix this (#108).
daverigby
added a commit
that referenced
this issue
Jun 17, 2024
Add progress bars for each of the main phases of an experiment -
Setup, Populate and Run. For Populate and Run we require the total
number of records / queries to show a useful end progress so far.
For the Populate phase we include the current rate of upsert. For the
Run phase we include the current latency and recall(*) values, these
require additional metrics of how many records have been upserted so
far (note that we generally upsert in batches, so the existing number
of Population requests is not sufficient.
(*) For recall we cannot calculate the current metric (last 10s), as
we only have a single histogram accumulating the results - instead
this shows the overall recall so far. There's an improvement
raised to fix this (#108).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We now report the current latency and recall during the run phase - e.g.
The current latency figures come from locust itself, which calcualates this from the last 10s worth of samples - so it is indeed a "current" metric.
However the recall figures are recorded by VSB itself from a single HdrHistogram instance, and hence they are the cumulative recall values so far.
This is misleading, as they won't readily adopt to changes to the performance of recall during the experiment.
To address this we need to do something similar to
_cache_response_times- retain N copies of the histogram from the last N seconds and use the difference between T=now and T-10 to display them.The text was updated successfully, but these errors were encountered: