Coordinated Omission correction

[alexaverbuch]

When a delay to execute an operation on time occurs:

• calculate how long it was delayed
• calculate the operations that would have been executed in that time
  • based on the workload mix
  • based on interval between operations
  • could do this by creating a (small) sample of the full workload during driver initialization, creating a looping iterator over that sample, then injecting/logging-to-metrics "fake" operations from this iterator, to account for the "omitted" operations
    • may need multiple samples from different workload timeframes to account for workloads that change over time
    • based on which operation failed, choose appropriate sample from appropriate timeframe
    • need to estimate scheduledStartTime, actualStartTime (perhaps equal to scheduledStartTime or perhaps equal to mean from previous operations?), runDuration (taken from mean of previous operations, for example)
    • would means from previous operations include the times from the injected/fake operations?

Alternatively, when a delay to execute an operation on time occurs:

• rather than executing that operation, send it to a "missed operations" place
• do not execute it
• do not log the operation in any other way
  tolerate up to some amount of such operations
• writes would still have to be executed, but reads do not need to be
• upon completion of the benchmark assign a some start and finish time to all "missed operations", based on means/percentiles/whatever of other operations of the same type/that were executed at the same period/whatever. place these in "corrected operations"
• include metrics from "corrected operations" to the final report metrics

Could possibly do this from a Delay Policy implementation

If the metrics logging service has this looping iterator the "fake" operations would not need to all get sent over a queue to the logging thread. Instead, just 1 message, containing a parameter communicating "number of fake operations to create", would be sent to the logging thread and it would deal with the rest

Make configurable:

• terminate
• correct from sample
• something else?
• ...