Unpin selenium to fix CI flakiness against current Chrome - #3955
Merged
Conversation
The testing requirements capped selenium at <=4.2.0 (2022), which predates Selenium Manager. CI installs the current stable Chrome (now 151) via an unpinned browser-actions/setup-chrome, and selenium 4.2 cannot reliably provision or drive it, producing scattered StaleElementReferenceException / TimeoutException failures across unrelated browser integration tests on every push and PR. Require selenium>=4.11.0 (mature Selenium Manager auto-provisions a matching chromedriver) up to the current latest 4.46.0.
Unpinning selenium exposed two deterministic breaks the 4.2.0 cap had hidden: - browser.py set the 'marionette' Firefox capability, which modern selenium/geckodriver reject with InvalidArgumentException (marionette is the implicit, only protocol now). Removed it. - Three test modules used the find_element(s)_by_* helper methods that selenium removed in 4.3. Migrated them to find_element(s)(By.*, ...).
Contributor
Author
|
Follow-up commit
Remaining |
selenium 4.3 changed move_to_element_with_offset to measure the offset from the element's center instead of its top-left corner. The dash_duo drag/click helpers (click_at_coord_fractions, zoom_in_graph_by_ratio) and the dcc page object helpers passed top-left-based fractional offsets (width*fx, height*fy), so under modern selenium they overshot past the element edge and raised MoveTargetOutOfBoundsException — failing the slider drag/step tests and the graph tooltip center-hover test. Convert the proportional offsets to center-relative (width*(fx-0.5)) and cast to int (W3C actions require integer pixels). Small fixed-pixel offsets (5, 8) are left as-is: they stay within any element regardless of origin.
dash_duo's _wait_for helpers raise selenium's TimeoutException(str(message)). Modern selenium's WebDriverException.__init__ calls super().__init__() with no args, so the message lives on .msg and .args is empty — test_duo's err.value.args[0] assertions raised IndexError. Read .msg, selenium's stable message accessor.
The step backgrounded Xvfb with a bare '&', so it inherited the step's stdout/stderr pipe to the Actions runner. Xvfb never exits, so that pipe never reached EOF and the runner blocked on the step indefinitely (intermittent 'Setup virtual display' hangs across the browser-test jobs). Redirect Xvfb's output to /dev/null and disown it so the step's pipe closes and the step completes immediately.
The redirect/disown alone did not stop the hang: the real culprit is
'apt-get update && apt-get install -y xvfb', which intermittently blocks on the
runner's dpkg/apt lock (apt-daily / unattended-upgrades). xvfb is already
preinstalled on the GitHub Ubuntu runners ('xvfb is already the newest
version'), so the install is pure risk. Just start the preinstalled Xvfb; if it
were ever absent the step fails fast instead of hanging.
Contributor
Author
CI verified — systemic flakiness fixedFull run + targeted re-runs of the flaky jobs. The two structural causes of the cross-PR flakiness are resolved:
Remaining reds are pre-existing flakes, not regressionsOn re-run, the DCC and Main Dash failures ( Recommend merging this PR (it strictly improves CI) and tracking the async-callback test failure + the racy per-test flakes in a separate issue. |
test_(async_)cbsc001/cbsc008 assert an exact one-callback-per-keystroke count, but the renderer coalesces same-identity callbacks still queued in its 'requested' state (requestedCallbacks.ts) into a single request. Two keystrokes landing in that batching window collapse into one invocation, so the count undershoots. The Lock choreography the tests used to serialize typing no longer holds now that async callbacks execute concurrently, and React 19's more aggressive event batching plus faster Chrome typing pushed the failure rate to ~90% locally — routinely exhausting the flaky retries. Gate each keystroke on the previous callback having executed (wait until the counter reflects it) so a keystroke's callback always leaves the 'requested' queue before the next is sent and can never be coalesced. This makes the exact-count assertion correct by construction; drop the Lock, the per-keystroke sleeps, and the @flaky retries.
…meout Two changes so a stuck test/server can no longer hang a whole CI step (the 'Run Async Callback Tests' step was wedging for the full job timeout): - ThreadedRunner.stop() Flask path called self.thread.join() with no timeout. If the injected SystemExit fails to unwind a worker stuck in a C call, that join blocks teardown forever. Bound it with stop_timeout (FastAPI and Quart paths already join with a timeout); the following until_not then fails fast instead of hanging. - Add pytest-timeout (requirements/ci.txt, installed via the [ci] extra in every test job) and set a 180s per-test cap in pytest.ini. Any remaining hang now fails with a full thread stack dump naming the test, instead of stalling the step until the job-level timeout.
Contributor
Author
✅ Full green runRun 32283425251 passed with zero failed jobs after the anti-hang + gating fixes. The two systemic problems are gone:
Remaining occasional reds ( Optional follow-up (not in this PR): add |
test_tdrp004_navigate_selected_cells read the derived-prop display cells with one-shot find_element().get_attribute() while keystrokes were still firing. props_container re-renders wholesale on every table-prop change, so the element went stale between find and read, failing Table Group 1 consistently once selenium was unpinned. Add a wait_prop() helper that re-finds the element each poll and waits for the value to settle, and use it for the tab-navigation assertions.
grbs007: the clickData callback also fires on load with clickData=None, setting the textarea to "null". The test read the value right after the click and raced that initial value - data != "" passed but json.loads returned None. Wait for the real click payload before parsing. dvcv003: the devtools error overlay intermittently reports an empty error title under React 19; mark it @flaky(max_runs=3), matching the existing convention for these overlay tests in this file (dvcv013).
test_rdcap003_side_effect_regression clicked #a and then counted the checklist options synchronously, racing the opts callback that re-renders them - so it read the previous count (assert 2 == 3). @flaky did not help because a slow runner loses the race on every rerun. Poll for the expected option count instead.
msmh003: counted the re-highlighted <span>s synchronously after the click, racing the callback-driven markdown swap (assert 2 == 3). Poll for the new span count. msps001: #dropdownsingle typed "one" + Enter with no wait, so Enter could fire before the list filtered and nothing was selected - the field then persisted as null. Wait for the filtered option before pressing Enter, mirroring the #dropdownmulti path.
test_arb008_set_props_chain_cb clicked #generated-button via wait_for_element().click(); the button re-renders as its n_clicks updates, so the handle went stale between find and click and threw StaleElementReferenceException - failing all 3 @flaky reruns across multiple runs once selenium was unpinned. Re-find and retry the click until it lands.
camdecoster
approved these changes
Aug 21, 2026
camdecoster
left a comment
camdecoster
left a comment
Contributor
There was a problem hiding this comment.
Looks fine. Would it be worth pinning Chrome to keep a stable test environment?
Co-authored-by: Cameron DeCoster <cameron.decoster@gmail.com>
Contributor
Author
I'd rather not pin chrome, when it's pinned it accumulates regressions we don't see on tests and it takes longer to fix afterwards (This took 3 days because selenium was pinned for so long). |
dveh002: read #output synchronously after 3 clicks, racing the callback and seeing the initial 'button clicks: 0'. Poll with wait_for_text_to_equal. msps001: clicked #dropdownmulti while the #dropdownsingle menu overlay was still closing, so the click was intercepted. Wait for the single dropdown's menu to close first.
empt001: counted table rows synchronously right after clicking clear, racing the callback (assert 3 == 0). Poll for the rows to drop. msps001: select #dropdownsingle by clicking the filtered option instead of pressing Enter - Enter did not reliably close the menu under load, leaving an overlay that intercepted the next dropdown's click.
The background-callbacks job repeatedly wedged its step for the full 15-min timeout even though every test passed: after pytest prints its summary, the suite leaves non-daemon workers (celery/diskcache/lingering servers) that block interpreter shutdown. pytest-timeout only bounds individual tests, not this post-session shutdown. Add a trylast pytest_sessionfinish hook (tests/conftest.py) that hard-exits via os._exit once the session is done - after the junit report is written and with the real exit status preserved - gated by DASH_TEST_FORCE_EXIT so local runs and other jobs are unaffected. Set that env on the background-callbacks job.
test_ddso002 drove keyboard navigation with ActionChains send_keys, which target document.activeElement; right after the menu opens the search input may not have focus yet, so ARROW_DOWN/SPACE were dropped and no option was selected (wait_for_text timed out). Send the keys to the .dash-dropdown-search element directly, which selenium focuses first.
ddso002: sending nav keys straight to the search input broke selection deterministically (SPACE typed a space instead of selecting). Restore the ActionChains approach and mark @flaky(max_runs=3) instead - the real flake is menu-input focus lagging menu-open, which needs the focused activeElement. rdmo002: the 'with lock' gating did not stop the renderer coalescing queued callbacks, so call_count came up short of 7. Gate each keystroke on its callback landing (wait.until) before sending the next.
test_rdps008 counted .column-header--delete synchronously right after clicking #deletable, racing the callback that re-renders the table (assert 1 == 0). Poll for the expected count with wait.until.
test_a11y006 and test_a11y008 drive keyboard navigation with ActionChains send_keys, which target document.activeElement; right after the menu opens the input may not have focus yet, dropping a keystroke (timeout, or landing on Option 2 instead of Option 3). Same class as ddso002 - sending keys to the input element instead breaks selection, so retry with @flaky(max_runs=3).
The threaded test runner handed out ports from a monotonic counter without checking availability. A previous test's server can linger on its port for a moment after teardown, so reusing that number failed with 'address already in use'; the test then hung until the per-test timeout (seen intermittently in the async/background suites). Probe each candidate port and skip any still bound before starting the server.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Recent pushes and PRs have been going red intermittently across the browser-based integration tests. The failures are scattered across unrelated Selenium tests (
test_persistence,test_csp,test_multi_output,test_derived_props, async callbacks, table server tests…), a different subset each run, all surfacing asStaleElementReferenceException/TimeoutException. That pattern is environmental flakiness, not a bad merge (which would fail the same test deterministically).Root cause
requirements/testing.txtpinnedselenium>=3.141.0,<=4.2.0(selenium 4.2.0 is from 2022).browser-actions/setup-chrome@v1withchrome-version: stable— unpinned — so CI now installs Chrome 151.Two amplifiers: the dependabot pip bump that would have raised selenium never landed on
dev, and the recent React 18/19 test matrix roughly doubled the browser shards, so a single flake reddens the whole run more often.Fix
Bump the pin to
selenium>=4.11.0,<=4.46.0. The>=4.11.0floor guarantees a mature Selenium Manager that auto-provisions a chromedriver matching whatever stable Chrome CI installs (this is whyinstall-chromedriver: falsein the setup step remains correct).Compatibility checks
find_element_by_*APIs anywhere indash/(those were dropped in selenium 4.3).webdriver.Chrome(options=...)/webdriver.Remote(command_executor=..., options=...).dash/testing/browser.pyuses, plus Selenium Manager availability, against selenium 4.46.0.Follow-ups (not in this PR)
test_async_cbsc001_simple_callback) and may be genuinely broken rather than flaky — worth a targeted look once this settles the noise.stablefine either way.