Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase number of writes in concurrency test to ensure conflict #6641

Merged

Conversation

eddyashton
Copy link
Member

We've seen a single failure of this test when backporting a recent locking change.

https://dev.azure.com/MSRC-CCF/CCF/_build/results?buildId=87860&view=logs&j=b6a3fffa-f95e-580b-9bcb-4ae7d4681578&t=7439823f-f6a2-5514-9be9-467f4c602fdd&l=145

test 6
      Start  6: kv_test

6: Test command: /CCF/build/kv_test
6: Environment variables: 
6:  TSAN_OPTIONS=suppressions=/CCF/tsan_env_suppressions
6:  ASAN_OPTIONS=alloc_dealloc_mismatch=0
6: Test timeout computed to be: 10000000
6: [doctest] doctest version is "2.4.11"
6: [doctest] run with "--help" for options
6: ===============================================================================
6: ../src/kv/test/kv_contention.cpp:253:
6: TEST SUITE: concurrency
6: TEST CASE:  get_version_of_previous_write ordering
6: 
6: ../src/kv/test/kv_contention.cpp:334: ERROR: CHECK( conflict_count > 0 ) is NOT correct!
6:   values: CHECK( {?} >  0 )
6: 
6: 2024-11-12T17:52:56.497726Z        100 [fail ] ../src/kv/committable_tx.h:253       | Error during serialisation
6: 2024-11-12T17:52:56.585602Z        100 [info ] ../src/kv/test/kv_test.cpp:3013      | Seed: 172319737
6: ===============================================================================
6: [doctest] test cases:   57 |   56 passed | 1 failed | 0 skipped
6: [doctest] assertions: 2901 | 2900 passed | 1 failed |
6: [doctest] Status: FAILURE!
 6/59 Test  #6: kv_test ..........................***Failed    7.50 sec

We expected a conflict, but saw none. Local runs produce hundreds of conflicts. Can repro this by pinning the test to a single core:

$ taskset -c 0 ./kv_test 
[doctest] doctest version is "2.4.11"
[doctest] run with "--help" for options
2024-11-13T10:32:26.372950Z        100 [info ] ../src/kv/test/kv_contention.cpp:336 | Found 0 conflicts
===============================================================================
../src/kv/test/kv_contention.cpp:253:
TEST SUITE: concurrency
TEST CASE:  get_version_of_previous_write ordering

../src/kv/test/kv_contention.cpp:337: ERROR: CHECK( conflict_count > 0 ) is NOT correct!
  values: CHECK( {?} >  0 )

2024-11-13T10:32:26.471539Z        100 [fail ] ../src/kv/committable_tx.h:253       | Error during serialisation
2024-11-13T10:32:26.563254Z        100 [info ] ../src/kv/test/kv_test.cpp:3013      | Seed: 718927790
===============================================================================
[doctest] test cases:    57 |    56 passed | 1 failed | 0 skipped
[doctest] assertions: 14372 | 14371 passed | 1 failed |
[doctest] Status: FAILURE!

So maybe, on the 4-core ACI boxes, we don't run for long enough to see any actual conflicts. This mostly just bumps up the number of writes.

Raised #6640 as a sensible follow-up.

@eddyashton eddyashton requested a review from a team November 13, 2024 10:44
@achamayou achamayou added this pull request to the merge queue Nov 13, 2024
@achamayou achamayou added auto-backport Automatically backport this PR to LTS branch 5.x-todo PRs which should be backported to 5.x labels Nov 13, 2024
@achamayou
Copy link
Member

@eddyashton we should backport this to 5.x to make sure the test is stable there too.

Merged via the queue into microsoft:main with commit b52f1ea Nov 13, 2024
29 checks passed
@achamayou achamayou deleted the increase_conflict_test_iterations branch November 13, 2024 13:26
ghost pushed a commit that referenced this pull request Nov 13, 2024
@ghost ghost added the backported This PR was successfully backported to LTS branch label Nov 13, 2024
achamayou pushed a commit that referenced this pull request Nov 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5.x-todo PRs which should be backported to 5.x auto-backport Automatically backport this PR to LTS branch backported This PR was successfully backported to LTS branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants