Skip to content

fix: add retries for flaky integ tests that depend on LLM responses#351

Merged
Hweinstock merged 1 commit intoaws:mainfrom
Hweinstock:fix/flaky-integ-test-retries
Mar 17, 2026
Merged

fix: add retries for flaky integ tests that depend on LLM responses#351
Hweinstock merged 1 commit intoaws:mainfrom
Hweinstock:fix/flaky-integ-test-retries

Conversation

@Hweinstock
Copy link
Contributor

@Hweinstock Hweinstock commented Mar 17, 2026

Issue #, if available: N/A

Description of changes:

Problem

Integration tests in test_session_manager.py that assert on LLM response content are inherently non-deterministic. A model might rephrase or omit expected keywords on any given run,
causing spurious failures that erode CI trust.

Solution

Added pytest-rerunfailures as a dev dependency and gave each potentially flaky test up to 3 total attempts
with a 5-second cooldown. Only the non-deterministic tests are marked; all other tests remain strict single-pass.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@codecov-commenter
Copy link

codecov-commenter commented Mar 17, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@cd2f2a0). Learn more about missing BASE report.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #351   +/-   ##
=======================================
  Coverage        ?   90.92%           
=======================================
  Files           ?       43           
  Lines           ?     4112           
  Branches        ?      634           
=======================================
  Hits            ?     3739           
  Misses          ?      204           
  Partials        ?      169           
Flag Coverage Δ
unittests 90.92% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Hweinstock Hweinstock marked this pull request as ready for review March 17, 2026 16:53
@Hweinstock Hweinstock requested a review from a team March 17, 2026 16:53
@Hweinstock Hweinstock merged commit 8424c7a into aws:main Mar 17, 2026
21 checks passed
@Hweinstock Hweinstock deleted the fix/flaky-integ-test-retries branch March 17, 2026 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants