Repository navigation
Commit 4426ccd
authored
fix(bigtable): don't notify config listeners while holding the manager lock (#14294)
Fixes #14177
Fixes #13903
## The bug
`ClientConfigurationManager.sendRequestWithRetries()` was `synchronized`
and registered its `whenComplete()` continuation **while still holding
the manager's monitor**.
When the `GetClientConfiguration` response lands before the continuation
is registered, `CompletableFuture` runs that continuation **inline on
the registering thread**. So `setClientConfiguration()` ->
`notifyListeners()` invoked alien listener code under the monitor. Any
thread that holds a listener's own lock and then calls a `synchronized`
manager method (e.g. `getClientConfiguration()`) closes the cycle and
deadlocks.
Both linked issues are this same bug: in each CI log the output stops
immediately after `testSysPropOverrideSessionLoadZero`'s `Local client
config override: session_configuration { }` line — i.e. entering
`testDeadlockPrevention` — and the job then hangs until it is cancelled
hours later (~6h in #13903, ~100min in #14177).
Evidence:
- **Deterministic reproduction.** `ThreadMXBean.findDeadlockedThreads()`
returns two threads; the main thread shows `- locked
ClientConfigurationManager@... at depth 11` inside
`sendRequestWithRetries` while blocked acquiring the listener's lock,
and the other thread is `BLOCKED` on the manager monitor inside
`getClientConfiguration()`.
- **Occurs naturally.** Over a real loopback gRPC channel under CPU
contention, 4 / 2000 iterations notified listeners with
`Thread.holdsLock(manager) == true`, clustered during JIT warmup.
## Why it hangs forever instead of timing out
The class already had `@Timeout(60s)`, but JUnit 5's default
`SAME_THREAD` mode enforces timeouts by *interrupting* the test thread,
and a thread blocked on monitor entry cannot be interrupted. The timeout
was a no-op, so the deadlock wedged the whole surefire JVM until the CI
job itself was cancelled.
## The fix
Narrow the lock so it covers only `sendRequest()`; completing the future
and registering the continuation now happen outside it.
This does not widen any race. The continuation was already required to
be correct without the lock, since it runs lock-free whenever the
response arrives on a gRPC thread — `setClientConfiguration()`,
`handleFailedFetch()` and `notifyListeners()` each take the monitor
themselves in narrow blocks. The change removes an *accidental* lock
hold, not a required one.
## Test changes
- **New deterministic regression test.** `testDeadlockPrevention` relies
on a real RPC and only hits the bad interleaving by luck. The new test
installs a `ClientInterceptor` that answers `GetClientConfiguration`
synchronously on the calling thread, so the future is always already
complete when the continuation is registered, and asserts
`Thread.holdsLock(manager)` is `false` in the callback. With the product
change reverted it fails every run.
- **`@Timeout` now uses `ThreadMode.SEPARATE_THREAD`** for this class,
so a future regression fails in seconds instead of wedging the runner.
Verified: against the unfixed manager, a deadlocking test now fails in
~5s with `TimeoutException` rather than hanging. Scoped to this class
rather than set repo-wide, to avoid changing execution semantics for
unrelated tests.
- **Re-enabled `testDeadlockPrevention`**, which had been `@Disabled`
for #13903.
## Verification
- `mvn -pl google-cloud-bigtable test`: **2446 tests, 0 failures, 0
errors, 1 skipped**; `ClientConfigurationManagerTest` 11/11.
- `mvn -pl google-cloud-bigtable fmt:check`: clean (644 files, 0
non-complying).
- Negative control: reverting the product change makes the new test fail
deterministically.1 parent a03ee10 commit 4426ccd
2 files changed
Lines changed: 127 additions & 6 deletions
File tree
- java-bigtable/google-cloud-bigtable/src
- main/java/com/google/cloud/bigtable/data/v2/internal/util
- test/java/com/google/cloud/bigtable/data/v2/internal/util
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
286 | 286 | | |
287 | 287 | | |
288 | 288 | | |
289 | | - | |
| 289 | + | |
290 | 290 | | |
291 | | - | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
292 | 308 | | |
293 | 309 | | |
294 | 310 | | |
295 | 311 | | |
296 | | - | |
297 | 312 | | |
298 | 313 | | |
299 | 314 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
66 | 67 | | |
67 | 68 | | |
| 69 | + | |
68 | 70 | | |
69 | 71 | | |
70 | | - | |
71 | 72 | | |
72 | 73 | | |
| 74 | + | |
73 | 75 | | |
74 | 76 | | |
75 | 77 | | |
| |||
79 | 81 | | |
80 | 82 | | |
81 | 83 | | |
82 | | - | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
83 | 90 | | |
84 | 91 | | |
85 | 92 | | |
| |||
380 | 387 | | |
381 | 388 | | |
382 | 389 | | |
383 | | - | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
384 | 447 | | |
385 | 448 | | |
386 | 449 | | |
| |||
552 | 615 | | |
553 | 616 | | |
554 | 617 | | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
555 | 661 | | |
556 | 662 | | |
557 | 663 | | |
| |||
0 commit comments