Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump CockroachDB to 22.2.17 #3301

Merged
merged 2 commits into from
Jan 19, 2024
Merged

Bump CockroachDB to 22.2.17 #3301

merged 2 commits into from
Jan 19, 2024

Conversation

roger2hk
Copy link
Contributor

Fixes #3300.

Checklist

@roger2hk roger2hk changed the title Bump CockroachDB to 23.1.13 Bump CockroachDB to 22.2.17 Jan 18, 2024
@JAORMX
Copy link
Collaborator

JAORMX commented Jan 18, 2024

=== RUN   TestLogSuite/DequeueLeavesNoneQueued
W0118 01:18:25.752295    3636 tree_storage.go:375] TX commit error: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/474/2/1): "sql txn" meta={id=392f48e9 key=/Table/421/1/6041628873873342746/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.01106136 epo=0 ts=1705540702.707518406,2 min=1705540701.649429217,0 seq=3000} lock=true stat=PENDING rts=1705540701.649429217,0 wto=false gul=1705540702.149429217,0, stack:
goroutine 50 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.21.6/x64/src/runtime/debug/stack.go:24 +0x5e
github.com/google/trillian/storage/crdb.(*treeTX).Commit(0xc000201080, {0xd36db0, 0x123c280})
	/home/runner/work/trillian/trillian/storage/crdb/tree_storage.go:375 +0x22d
github.com/google/trillian/storage/crdb.(*crdbLogStorage).QueueLeaves(0xd38328?, {0xd36db0, 0x123c280}, 0x0?, {0xc00099c000, 0x3e8, 0x500}, {0x0, 0xecfb6864b, 0x0})
	/home/runner/work/trillian/trillian/storage/crdb/log_storage.go:301 +0x225
github.com/google/trillian/storage/crdb.TestQueueLeavesDuplicateBigBatch(0xc000b56340)
	/home/runner/work/trillian/trillian/storage/crdb/log_storage_test.go:246 +0x250
testing.tRunner(0xc000b56340, 0xc74738)
	/opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1648 +0x3ad
=== NAME  TestQueueLeavesDuplicateBigBatch
    log_storage_test.go:247: Failed to queue leaves: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/474/2/1): "sql txn" meta={id=392f48e9 key=/Table/421/1/6041628873873342746/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.01106136 epo=0 ts=1705540702.707518406,2 min=1705540701.649429217,0 seq=3000} lock=true stat=PENDING rts=1705540701.649429217,0 wto=false gul=1705540702.149429217,0
--- FAIL: TestQueueLeavesDuplicateBigBatch (4.95s)

🤔

@roger2hk
Copy link
Contributor Author

=== RUN   TestLogSuite/DequeueLeavesNoneQueued
W0118 01:18:25.752295    3636 tree_storage.go:375] TX commit error: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/474/2/1): "sql txn" meta={id=392f48e9 key=/Table/421/1/6041628873873342746/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.01106136 epo=0 ts=1705540702.707518406,2 min=1705540701.649429217,0 seq=3000} lock=true stat=PENDING rts=1705540701.649429217,0 wto=false gul=1705540702.149429217,0, stack:
goroutine 50 [running]:
runtime/debug.Stack()
	/opt/hostedtoolcache/go/1.21.6/x64/src/runtime/debug/stack.go:24 +0x5e
github.com/google/trillian/storage/crdb.(*treeTX).Commit(0xc000201080, {0xd36db0, 0x123c280})
	/home/runner/work/trillian/trillian/storage/crdb/tree_storage.go:375 +0x22d
github.com/google/trillian/storage/crdb.(*crdbLogStorage).QueueLeaves(0xd38328?, {0xd36db0, 0x123c280}, 0x0?, {0xc00099c000, 0x3e8, 0x500}, {0x0, 0xecfb6864b, 0x0})
	/home/runner/work/trillian/trillian/storage/crdb/log_storage.go:301 +0x225
github.com/google/trillian/storage/crdb.TestQueueLeavesDuplicateBigBatch(0xc000b56340)
	/home/runner/work/trillian/trillian/storage/crdb/log_storage_test.go:246 +0x250
testing.tRunner(0xc000b56340, 0xc74738)
	/opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1595 +0xff
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.21.6/x64/src/testing/testing.go:1648 +0x3ad
=== NAME  TestQueueLeavesDuplicateBigBatch
    log_storage_test.go:247: Failed to queue leaves: pq: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to a conflict: committed value on key /Table/3/1/474/2/1): "sql txn" meta={id=392f48e9 key=/Table/421/1/6041628873873342746/"\x00\xe1\t\xcf\\#\xd6'U\x1e\xaf\xa6\xed\x82y\xb6.JIGn\x83Cxy\xf6\x85\xa0\xe01\x91P"/0 pri=0.01106136 epo=0 ts=1705540702.707518406,2 min=1705540701.649429217,0 seq=3000} lock=true stat=PENDING rts=1705540701.649429217,0 wto=false gul=1705540702.149429217,0
--- FAIL: TestQueueLeavesDuplicateBigBatch (4.95s)

🤔

As go test runs concurrently and both of the crdb testservers now use the same version, I doubt if both of testservers are sharing the same data file behind the scene causing the unexpected error.

@roger2hk roger2hk marked this pull request as ready for review January 18, 2024 18:39
@roger2hk roger2hk requested a review from a team as a code owner January 18, 2024 18:39
@roger2hk
Copy link
Contributor Author

The default configuration of testserver stores the data in memory. I suspected that's the root cause failing the tests when two testservers are concurrently reading/writing in the same memory database. Thus, I've added testserver.StoreOnDiskOpt() into the latest commit to store the test data in two different on-disk database. The database is saved at /tmp/cockroach-testserverxxxxxxxx, with randomly generated xxxxxxxx postfix.

@JAORMX
Copy link
Collaborator

JAORMX commented Jan 19, 2024

@roger2hk that makes sense. Thanks for taking a look!

@roger2hk roger2hk merged commit 2f829cf into google:master Jan 19, 2024
10 checks passed
@roger2hk roger2hk deleted the fix-crdb-test branch January 19, 2024 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CRDB Check Broken due to missing the latest (v23.1.14) binary
3 participants