What happened?
The pyamber test TestCleanupFailedUpload::test_delete_object_failure_is_swallowed fails intermittently on CI. It is timing/GC-dependent, so it passes locally but trips on CI runners under load.
The test expects delete_object to be called once after a simulated upload failure, but on the failing run it's called twice:
AssertionError: Expected 'delete_object' to be called once. Called 2 times.
Root cause is cross-test state leakage, not a Python version regression. The preceding test, test_write_after_upload_error_raises_error, creates a LargeBinaryOutputStream, triggers an upload-thread failure, and does not close the stream. When that leaked stream is garbage-collected, IOBase.del invokes our close(), which calls _cleanup_failed_upload, which does a fresh large_binary_manager._get_s3_client() lookup. If GC happens to fire while test_delete_object_failure_is_swallowed's patch.object(...) is active, the leaked stream's cleanup invokes delete_object on that test's mock_s3. The victim test then runs its own close() and calls delete_object a second time on the same mock assert_called_once_with fails.
This blocks CI nondeterministically on PRs (observed on a docs-only branch that touches no Python code).
How to reproduce?
Push any branch and let CI run — the failure is timing-dependent so it does not reproduce every time, but it shows up under runner load: https://github.com/apache/texera/actions/runs/26481776334/job/77980417021
To reproduce deterministically locally, force GC between the two tests so the leaked stream finalizes while the next test's patches are active:
After the leaky test runs, inside the next test, before any assertions:
import gc; gc.collect()
Branch
main
Commit Hash (Optional)
8a7366f
What browsers are you seeing the problem on?
No response
Relevant log output
FAILED test_large_binary_output_stream.py::TestCleanupFailedUpload::test_delete_object_failure_is_swallowed
AssertionError: Expected 'delete_object' to be called once. Called 2 times.
Calls: [call(Bucket='test-bucket', Key='path/to/object'),
call(Bucket='test-bucket', Key='path/to/object')].
Runner: /opt/hostedtoolcache/Python/3.12.13/x64
What happened?
The pyamber test TestCleanupFailedUpload::test_delete_object_failure_is_swallowed fails intermittently on CI. It is timing/GC-dependent, so it passes locally but trips on CI runners under load.
The test expects delete_object to be called once after a simulated upload failure, but on the failing run it's called twice:
AssertionError: Expected 'delete_object' to be called once. Called 2 times.
Root cause is cross-test state leakage, not a Python version regression. The preceding test, test_write_after_upload_error_raises_error, creates a LargeBinaryOutputStream, triggers an upload-thread failure, and does not close the stream. When that leaked stream is garbage-collected, IOBase.del invokes our close(), which calls _cleanup_failed_upload, which does a fresh large_binary_manager._get_s3_client() lookup. If GC happens to fire while test_delete_object_failure_is_swallowed's patch.object(...) is active, the leaked stream's cleanup invokes delete_object on that test's mock_s3. The victim test then runs its own close() and calls delete_object a second time on the same mock assert_called_once_with fails.
This blocks CI nondeterministically on PRs (observed on a docs-only branch that touches no Python code).
How to reproduce?
Push any branch and let CI run — the failure is timing-dependent so it does not reproduce every time, but it shows up under runner load: https://github.com/apache/texera/actions/runs/26481776334/job/77980417021
To reproduce deterministically locally, force GC between the two tests so the leaked stream finalizes while the next test's patches are active:
After the leaky test runs, inside the next test, before any assertions:
import gc; gc.collect()
Branch
main
Commit Hash (Optional)
8a7366f
What browsers are you seeing the problem on?
No response
Relevant log output
FAILED test_large_binary_output_stream.py::TestCleanupFailedUpload::test_delete_object_failure_is_swallowed AssertionError: Expected 'delete_object' to be called once. Called 2 times. Calls: [call(Bucket='test-bucket', Key='path/to/object'), call(Bucket='test-bucket', Key='path/to/object')]. Runner: /opt/hostedtoolcache/Python/3.12.13/x64