test(backup): Measure read/write latency during back up

[kreuzerkrieg]

RW latency during backup using rclone

Test requirements

• 50% reads and 50% writes.
• Compaction enabled.
• 75%-85% CPU utilization (see discussion)
• Expect up to 6-7ms P99 read latencies (see discussion)
• Expect not more than 10ms read latency during backup

Argus run
https://argus.scylladb.com/tests/scylla-cluster-tests/b6a76dc0-a6a8-47b8-9dc0-5eff576866e5
Results:

	P90 write [ms]	P90 read [ms]	P99 write [ms]	P99 read [ms]	Throughput write [op/s]	Throughput read [op/s]	duration [HH:MM:SS]	start time	Overview	QA dashboard
Cycle #​1	$${\color{green}1.71}$$	$${\color{green}1.89}$$	$${\color{red}1719.66}$$	$${\color{red}1737.49}$$	124,209	124,233	$${\color{green}00:10:14}$$	10:14:50	view	view
Cycle #​2	$${\color{green}0.60}$$	$${\color{green}0.69}$$	$${\color{green}5.51}$$	$${\color{green}5.74}$$	124,306	124,365	$${\color{green}00:10:14}$$	10:36:23	view	view

	backup time [s]
Backup	00:08:51
Backup during read stress	00:17:28

We can safely say that nor read neither write are affected by backup

@bhalevy / @regevran FYI

[mikliapko]

Added qa-mainteiners to Reviewers

[soyacz]

If you have expectations for the metrics (like p99's below 8ms) consider using ValidationRule(fixed_limit=8) similarly for other metrics. See sdcm.argus_results.ManagerRestoreBanchmarkResult for example.

[kreuzerkrieg]

If you have expectations for the metrics (like p99's below 8ms) consider using ValidationRule(fixed_limit=8) similarly for other metrics. See sdcm.argus_results.ManagerRestoreBanchmarkResult for example.

I strongly oppose this type of validation as it may cause more harm than good. Occasionally, factors like "noisy neighbors" can cause failures, leading people to spend days troubleshooting. Moreover, this validation is specific to a particular type of AWS instance. Using any other instance type will result in failure.

[roydahan]

See my comments about the stress commands.
+
It needs to have a jenkins pipeline.

[kreuzerkrieg]

See my comments about the stress commands. + It needs to have a jenkins pipeline.

What we are going to achieve by adding (clonning) a new pipeline?

[roydahan]

What did you base the "17000/s" on?
A typical max throughput test for mixed workload using i4i.4xlarge achieves around 500K op/s.
4 loaders running 17K op/s is equal to 68K op/s which is about 13% of the max throughput.
(BTW, to achieve that we use 620 threads and bigger loaders - you can probably start with c7i.2xlarge).

lastly, I think that 10m may be too short to get consistent results.

[kreuzerkrieg]

What did you base the "17000/s" on? A typical max throughput test for mixed workload using i4i.4xlarge achieves around 500K op/s. 4 loaders running 17K op/s is equal to 68K op/s which is about 13% of the max throughput. (BTW, to achieve that we use 620 threads and bigger loaders - you can probably start with c7i.2xlarge).

ok, then how do I calculate number of threads and op/s to get to 6-7ms latency?

lastly, I think that 10m may be too short to get consistent results.

how long it will be considered OK-ish?

[roydahan]

ok, then how do I calculate number of threads and op/s to get to 6-7ms latency?

Is your target to measure latency during the backup operation or to measure duration of the operation under a specific load?
If you want to measure latency, you don't know the latency :)
We usually measure the latency under a workload which is 50% of max throughput.
in your case it would be something like "-rate 'threads=620 fixed=62500/s' "

[roydahan]

how long it will be considered OK-ish?

How long does the backup operation take?

[kreuzerkrieg]

how long it will be considered OK-ish?

How long does the backup operation take?

10min

[roydahan]

The location of this should be probably under performance and not under manager.

[roydahan]

Which bring me to the question of, why didn't you just replicate the same test we have for latency_with_operations and replaced the operation with a manager backup nemesis.
If needed, you could just define new manager backup nemesis.

[regevran]

Please note that this test is to create a baseline, something to compare to when we run the same test but based on Scylla core backup capabilities.
I prefer focusing on making sure that this test is OK in what it tests and that it can be duplicated and changed to core-scylla test.

[soyacz]

@regevran see you can apply validation rules that will fail the test if value exceeds error threshold (configurable, see validation rules examples in other tests). In case of setting custom rules for latency_decorator_calculator, see example .

btw. after rebase you can remove mixed word from the test name and add workload_name: mixed to config.

[scylladbbot]

@kreuzerkrieg new branch branch-2025.1 was added, please add backport label if needed