Flaky: AuthProxyHeadersTest.rtm intermittently sees 3 proxy tunnels, expects 2

[zimeg]

Summary

test_with_remote_apis.AuthProxyHeadersTest.rtm is flaky. It asserts the embedded proxy handled exactly 2 connections:

assertThat(callCount.get(), is(2));

but intermittently observes 3 (Expected: is <2> ... but: was <3>), failing the build. The test's callCount is incremented by the embedded Jetty ConnectHandler on every non-407 CONNECT tunnel, so a 3 means an extra tunnel was opened through the proxy during the RTM flow.

Status of the attempted fix (this is the point of the issue)

A change on the api branch switched the call to rtmConnect(rtmBotToken, false) — dropping the fullUserInfoRequired users.info call on the theory that it was the extra HTTPS round-trip inflating the tunnel count to 3. See the change on api:

• commit 89bf7f3 — "test: stabilize AuthProxyHeadersTest.rtm proxy connection count"
• current test on api: AuthProxyHeadersTest.java

// two connections (rtm.connect + websocket), otherwise a third tunnel can be opened
try (RTMClient rtm = slack.rtmConnect(rtmBotToken, false)) {

That change does not make the test deterministic. A build that includes the false change on api still failed rtm with was <3> at the is(2) assertion. So dropping users.info was, at most, a partial mitigation — the extra CONNECT tunnel has another source (a WebSocket reconnect, a keep-alive revalidation opening a fresh tunnel, or similar timing). Whether the change lowers the flake rate is unconfirmed; it does not eliminate the flake.

Repro / evidence

Intermittent — the same test passes on other builds, and fails on others with the false change present. Confirmed by the fact that the assertion still hits was <3> on api, which carries the change.

Possible directions (not yet chosen)

1. Assert a bounded range instead of an exact count — the test's real intent is "the proxy was used with the auth header," not "exactly N tunnels." A tolerant assertion (e.g. >= 2, or anyOf(is(2), is(3))) would kill the flake without weakening what's verified (a 407 or a leaked connection would still show up as a wrong count). Follows the same pattern already used in the sibling scim() test in this file (assertTrue(callCount.get() >= 1)).
2. Find and eliminate the extra tunnel — trace whether the RTM WebSocket connect opens a second CONNECT under the proxy (reconnect / pooled-connection revalidation) and make the count deterministic. Harder; needs the proxy's per-CONNECT logging enabled to see the real tunnel sequence.

Filing so the flake is on the record and the false change on api isn't mistaken for a settled fix.

[zimeg]

Update: this is broader than rtm — it's a class-wide (and possibly wider) proxy-tunnel-count fragility

Another run surfaced the same failure shape on a different method in AuthProxyHeadersTest:

• AuthProxyHeadersTest.methods — Expected: is <1> ... but: was <2>

So it's not specific to rtm (which saw is(2)/was(3)). Multiple methods in this class assert an exact proxy CONNECT count (callCount), and any of them can see an extra tunnel when the underlying HTTP connection pool opens a spurious/duplicate CONNECT (keep-alive revalidation, reconnect, route re-establishment). Whichever method happens to run when that occurs fails on the off-by-one.

Implications:

• The rtmConnect(rtmBotToken, false) change on api was patching one symptom (rtm) on the theory of one specific extra call. Since the same off-by-one now shows on methods, that per-method tweak is too narrow — it can't cover the class.
• The robust fix is tolerant assertions across the class — assert the auth path was exercised (a lower-bound / bounded-range count) rather than an exact tunnel count, for every method in AuthProxyHeadersTest that checks callCount. This preserves what the test actually cares about (the proxy was used with the correct auth header; a 407 or a leaked/dropped connection still shows as a wrong count) while removing the pool-timing flake.
• More broadly: any test that asserts an exact count of proxy tunnels / pooled connections is inherently fragile against connection-pool nondeterminism. Worth treating as a pattern, not a one-off per method.

Reframing the issue title/scope accordingly: AuthProxyHeadersTest's exact proxy-connection-count assertions are flaky class-wide, not just rtm.