Skip to content

fix: Stop the FDv1 stream on the stream thread after an unrecoverable HTTP error - #402

Merged
kinyoklion merged 1 commit into
mainfrom
rlamb/sdk-3180/fdv1-stream-stop-on-unrecoverable
Sep 25, 2026
Merged

kinyoklion merged 1 commit into
mainfrom
rlamb/sdk-3180/fdv1-stream-stop-on-unrecoverable

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Summary

The FDv1 StreamingDataSource sometimes opens a second stream connection after an unrecoverable HTTP status (401, 403, 405, ...), even though it has decided to stop. The v2 contract tests streaming/retry behavior/do not retry after unrecoverable HTTP error on initial connect and .../on reconnect fail intermittently because of it, both in CI and locally against unmodified main (3 of 4 runs of that test group).

Cause. The decision to stop lives in the BackgroundEventHandler.onError callback. BackgroundEventSource dispatches that callback to its event executor and then, with no ConnectionErrorHandler configured, goes straight back to reading, which reconnects after the retry delay. The callback's stop() spawns yet another thread to close the source. Two thread hops versus a reconnect delay of 1 ms (the harness setting) is a race the reconnect usually wins; the default 1 s delay makes it rare in production but does not remove it.

Fix. Give the BackgroundEventSource a ConnectionErrorHandler that returns SHUTDOWN for a StreamHttpErrorException whose status is not recoverable per LDUtil.isHttpErrorRecoverable, and PROCEED otherwise. BackgroundEventSource consults it on the stream thread before the next reconnect attempt, so no second connection can be made. The onError bookkeeping (error callback, 401 guard, sink shutdown, stop()) is unchanged; recoverable statuses still retry with backoff.

Tests. Two new StreamingDataSourceTest cases use a 1 ms reconnect delay and a sequential server: an unrecoverable status on the initial connect, and one on the reconnect after a dropped stream. Each asserts that no further request arrives. Both fail on every run without the fix (3 of 3) and pass on every run with it (5 of 5).

Verified on an emulator with the v2.41.0 harness: the streaming/retry behavior group passed 6 of 6 runs against this branch (unmodified main failed 3 of 4), and the full v2 suite passes (848 total, 832 ran, exit 0). SDK unit tests: 732 pass.


Note

Overview
Fixes a race where FDv1 streaming could open another connection after an unrecoverable HTTP status (401, 403, etc.) even though onError had already decided to stop. Stop logic ran on the event executor while BackgroundEventSource could reconnect on the stream thread before stop() took effect—especially visible with a 1 ms retry delay in contract tests.

StreamingDataSource now configures a ConnectionErrorHandler on BackgroundEventSource that returns SHUTDOWN when the error is a StreamHttpErrorException with a status LDUtil.isHttpErrorRecoverable rejects, and PROCEED otherwise. That decision runs on the stream thread before the next reconnect, so a second request cannot win the race. Existing onError behavior (callbacks, 401 guard, sink shutdown, recoverable retries) is unchanged.

Tests add configurable initialReconnectDelayMillis in the test helper and two HTTP-server cases: unrecoverable status on initial connect and on reconnect after a successful stream—each asserts no further requests after the failure.

Reviewed by Cursor Bugbot for commit cfa8126. Bugbot is set up for automated code reviews on this repo. Configure here.

… HTTP error

The decision to stop after a 401, 403, or other unrecoverable status was
made in the onError callback, which runs on a different thread from the
stream reader. BackgroundEventSource reconnected after the retry delay
before that stop() took effect, so the SDK opened a second connection it
should not have made. A ConnectionErrorHandler now returns SHUTDOWN for
unrecoverable statuses; BackgroundEventSource checks it on the stream
thread before any reconnect.
@kinyoklion
kinyoklion marked this pull request as ready for review September 25, 2026 16:51
@kinyoklion
kinyoklion requested a review from a team as a code owner September 25, 2026 16:51
@kinyoklion
kinyoklion merged commit 0f40e44 into main Sep 25, 2026
8 checks passed
@kinyoklion
kinyoklion deleted the rlamb/sdk-3180/fdv1-stream-stop-on-unrecoverable branch September 25, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants