test(spanner): add pytest-xdist parallel execution with state isolation by chalmerlowe · Pull Request #17344 · googleapis/google-cloud-python

chalmerlowe · 2026-06-02T14:19:19Z

This PR adds pytest-xdist to parallelize unit tests and the core_deps_from_source nox sessions for the google-cloud-spanner package.

By running tests in parallel using -n auto, the execution time of the Spanner unit tests are reduced from ~8 minutes to 4 minutes.

State isolation and test reliability are achieved by:

Simplifying Subtests: We pass simple strings/names to subTest() instead of complex objects. This keeps subtests lightweight and prevents serialization errors.
Cleaning Up Global Singletons: We reset telemetry singleton states on test teardown (using pytest's idiomatic monkeypatch fixture). This ensures metric counters don't leak from one test into another.
Fixing Concurrent Mock Conflicts: We return fresh mocked iterators for concurrent calls (using side_effect instead of return_value). This prevents one thread from exhausting a mock's results before another thread can read them.
Robust Assertion Checks: We added a helper method (_assert_concurrent_transaction_invariants) that verifies the behavior of concurrent threads (ensuring exactly one thread starts the transaction while others wait/reuse it), rather than checking fragile call logs or counting sequential request IDs. This allowed us to safely run four concurrent tests.

Note

The long pole in the tent is still the system tests, which require about 30 minutes. It is not as simple as just adding xdist because there are other factors that limit velocity including the fact that system tests actually interact with live systems.

… type check

gemini-code-assist

Code Review

This pull request parallelizes unit tests by installing pytest-xdist and running pytest with -n auto, while addressing test isolation issues by resetting metrics singletons, cleaning up asyncio event loops, and updating subtests to use string identifiers. Feedback on these changes suggests keeping the mypy session skipped as it is out of scope, using pytest's monkeypatch fixture for environment variable management, and declaring pytest-xdist in the package's primary dependency file.

chalmerlowe · 2026-06-02T15:16:30Z


        return mock.create_autospec(SpannerClient, instance=True)

+    def _assert_concurrent_transaction_invariants(self, call_args_list, expected_count=2):


For the reviewer:
This new function is a helper that enables us to confirm test results with more stability in concurrently run systems by checking for elements that won't change during a concurrent transation rather than elements that might change depending on the order that a concurrent set of tests finishes.

There are four locations where we strip out some long test bodies that were very specific (e.g. did test 1.1 come before 2.1, etc) and replace them with:

Is there a single "begin" transaction.
Do other transactions reuse the same transaction ID throughout.

chalmerlowe · 2026-06-02T15:17:48Z


        self._batch_update_helper(transaction=transaction, database=database, api=api)

-        api.execute_sql.assert_any_call(


As noted above, here we are replacing the test body that was looking for specific id numbers (and became brittle when run in a concurrent transaction).

Replaced it with the more invariant test body from the helper function: _assert_concurrent_transaction_invariants

chalmerlowe · 2026-06-02T15:18:07Z

        self._execute_update_helper(transaction=transaction, api=api)
        self.assertEqual(api.execute_sql.call_count, 1)

-        api.execute_sql.assert_any_call(


Replaced the test body with the more invariant test body: _assert_concurrent_transaction_invariants

chalmerlowe · 2026-06-02T15:18:29Z

-        )
+        self._assert_concurrent_transaction_invariants(api.execute_batch_dml.call_args_list, 2)

-        self.assertEqual(api.execute_batch_dml.call_count, 2)


Replaced the test body with the more invariant test body: _assert_concurrent_transaction_invariants

chalmerlowe · 2026-06-02T15:18:41Z


        self._execute_update_helper(transaction=transaction, api=api)

-        api.execute_sql.assert_any_call(


Replaced the test body with the more invariant test body: _assert_concurrent_transaction_invariants

chalmerlowe · 2026-06-02T15:19:10Z


        self._execute_update_helper(transaction=transaction, api=api)

-        begin_read_write_count = sum(


Replaced the test body with the more invariant test body: _assert_concurrent_transaction_invariants

daniel-sanche

couple small comments, otherwise LGTM

daniel-sanche · 2026-06-02T20:04:13Z

            called_with.append((txn, args, kw))
            txn.insert(TABLE_NAME, COLUMNS, VALUES)

+        import threading


can this be moved to to top of the file?

daniel-sanche · 2026-06-02T20:04:32Z

            called_with.append((txn, args, kw))
            txn.insert(TABLE_NAME, COLUMNS, VALUES)

+        import threading


if we import threading globally, this shouldnt be needed

daniel-sanche · 2026-06-02T20:05:31Z

            called_with.append((txn, args, kw))
            txn.insert(TABLE_NAME, COLUMNS, VALUES)

+        import threading


same comments about imports

daniel-sanche · 2026-06-02T20:09:16Z

        CURRENT_DIRECTORY / "testing" / f"constraints-{session.python}.txt"
    )
    install_unittest_dependencies(session, "-c", constraints_path)
+    session.install("pytest-xdist")


Can this be added to UNIT_TEST_STANDARD_DEPENDENCIES?

* move 'threading' import to top of test_session.py files * move pytest-xdist to UNIT_TEST_STANDARD_DEPENDENCIES

This addresses a 401 error arising in background threads during pytest-xdist runs due to state leaks across the metrics exporter.

daniel-sanche

LGTM

feat(spanner): fix asyncio event loop leak in unit tests and activate…

b49caaf

… type check

gemini-code-assist Bot reviewed Jun 2, 2026

View reviewed changes

Comment thread packages/google-cloud-spanner/noxfile.py

Comment thread packages/google-cloud-spanner/tests/unit/conftest.py Outdated

Comment thread packages/google-cloud-spanner/noxfile.py Outdated

chalmerlowe force-pushed the chore/add-nox-session-google-cloud-spanner-xdist branch 3 times, most recently from 22b979b to cc28cd2 Compare June 2, 2026 15:07

chalmerlowe commented Jun 2, 2026

View reviewed changes

chalmerlowe force-pushed the chore/add-nox-session-google-cloud-spanner-xdist branch 7 times, most recently from 7821620 to ab88581 Compare June 2, 2026 17:08

test(spanner): add pytest-xdist parallel execution with state isolation

72804c4

chalmerlowe force-pushed the chore/add-nox-session-google-cloud-spanner-xdist branch from ab88581 to 72804c4 Compare June 2, 2026 17:25

chalmerlowe added this to the Update nox sessions milestone Jun 2, 2026

daniel-sanche reviewed Jun 2, 2026

View reviewed changes

chalmerlowe added 3 commits June 4, 2026 10:39

chore(spanner): address reviewer feedback

f7d138d

* move 'threading' import to top of test_session.py files * move pytest-xdist to UNIT_TEST_STANDARD_DEPENDENCIES

test(spanner): prevent background metrics exporter thread leaks

8a98679

This addresses a 401 error arising in background threads during pytest-xdist runs due to state leaks across the metrics exporter.

chore(spanner): fix formatting and lint errors

db07ea3

chalmerlowe marked this pull request as ready for review June 4, 2026 22:23

chalmerlowe requested a review from a team as a code owner June 4, 2026 22:23

daniel-sanche approved these changes Jun 4, 2026

View reviewed changes

chalmerlowe merged commit 761c805 into main Jun 4, 2026
32 checks passed

chalmerlowe deleted the chore/add-nox-session-google-cloud-spanner-xdist branch June 4, 2026 22:34

parthea mentioned this pull request Jun 5, 2026

chore: update googleapis and regenerate #17383

Merged

suztomo mentioned this pull request Jun 5, 2026

Regeneration check found diff #17373

Open


		return mock.create_autospec(SpannerClient, instance=True)

		def _assert_concurrent_transaction_invariants(self, call_args_list, expected_count=2):


		self._batch_update_helper(transaction=transaction, database=database, api=api)

		api.execute_sql.assert_any_call(


		self._execute_update_helper(transaction=transaction, api=api)

		api.execute_sql.assert_any_call(


		self._execute_update_helper(transaction=transaction, api=api)

		begin_read_write_count = sum(

Conversation

chalmerlowe commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chalmerlowe Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-sanche left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daniel-sanche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chalmerlowe commented Jun 2, 2026 •

edited

Loading

chalmerlowe Jun 2, 2026 •

edited

Loading

chalmerlowe Jun 2, 2026 •

edited

Loading

chalmerlowe Jun 2, 2026 •

edited

Loading

chalmerlowe Jun 2, 2026 •

edited

Loading

chalmerlowe Jun 2, 2026 •

edited

Loading