[BUG]: "redis-benchmark-spec" test suite failures

[odidev]

Describe the bug

I am executing benchmark tests following the documents here: < https://github.com/redis/redis-benchmarks-specification#testing-out-the-redis-benchmarks-spec-client-runner > on the Linux/AMD64 Ubuntu AWS instance.

However, tests failed with the below error:

2023-05-24 09:08:06 INFO Benchmark used memory at start of benchmark: 1g 
2023-05-24 09:08:06 INFO Checking if there is a keyspace check being enforced 
2023-05-24 09:08:06 INFO Ensuring keyspace length requirement = 1000000 is met. 
2023-05-24 09:08:06 ERROR The total numbers of keys in setup does not match the expected spec: 1000000!=1000002. Aborting... 
2023-05-24 09:08:06 CRITICAL Some unexpected exception was caught during local work. Failing test.... 
2023-05-24 09:08:06 CRITICAL <class 'Exception'> 

Referring to the already raised ticket here: < #206 >, I already installed the latest version of redis-benchmarks-specification (0.1.70) available with pip. But I am getting the same error.

Can you please provide some pointers for the same?

Environment (please complete the following information):

• OS: Ubuntu 22.04
• PyPi version package: pip 23.1.2 from /usr/local/lib/python3.10/dist-packages/pip (python 3.10)
• Platfrom: x86

[filipecosta90]

@odidev do you start with an empty DB before the test?
Can you retry with --flushall_on_every_test_start option enabled?

[odidev]

@filipecosta90 , thank you for the response. --flushall_on_every_test_start helped running tests as expected.

I ran some random tests on both the x86_64 and ARM64 AWS instances. Those tests are passing.
However, there is a build variant used in all YML files, as below:

build-variants:
- gcc:8.5.0-amd64-debian-buster-default

The build variant is hardcoded for AMD64 platform. Since tests are passing even on the ARM64 platform, may I know, what is the significance of this build variant here?