Skip to content

[python] Fix flaky Daft-on-Ray CI failures - #9975

Open
XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/python-ci-ray-shm
Open

XiaoHongbo-Hope wants to merge 1 commit into
apache:masterfrom
XiaoHongbo-Hope:codex/python-ci-ray-shm

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

The Daft-on-Ray end-to-end test intermittently exits with SIGSEGV on Python 3.13 after its assertions and Ray cleanup have completed. This makes otherwise unrelated Python CI runs fail nondeterministically.

Root cause

The Ray runner leaves Daft progress-bar objects alive until interpreter shutdown. While CPython is clearing module dictionaries, tqdm.__del__() calls close() and renders the final display, which constructs datetime.timedelta. The native crash occurs in delta_new() under slot_tp_finalize() / finalize_modules().

This is unrelated to the Docker /dev/shm size: the same failure reproduces locally with 31 GiB of shared memory.

Fix

Disable Daft's cosmetic progress bars only in the isolated, output-captured E2E subprocess by setting DAFT_PROGRESS_BAR=0. The test behavior and assertions are unchanged, and no product runtime setting is affected.

Validation

  • reproduced with Python 3.13.15, Daft 0.7.25, Ray 2.54.0, and PyArrow 18.1.0
  • captured a core dump showing _Py_Finalize()finalize_modules()slot_tp_finalize()tqdm.__del__()datetime.timedelta() / delta_new()
  • baseline failed on attempts 3 and 4 in repeated output-captured runs
  • fixed version passed 30/30 repeated output-captured runs
  • targeted pytest passed
  • git diff --check

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Give Ray tests sufficient shared memory [python] Fix flaky Daft-on-Ray CI failures Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant