Mntor 5319 - #6826
Draft
GregHilstonMozilla wants to merge 5 commits into
Draft
Conversation
redisClient() declared a module-level singleton but never read it, so every call built a new ioredis instance and abandoned the previous socket.
Sentry JS v10 removed the internal debug logger from @sentry/core and renamed it `debug`; the `logger` export now points at the Structured Logs API.
retryStrategy threw once times > 3. ioredis calls that callback outside any try/catch, so the throw escaped as an uncaught exception and took the process down, and a refused connection had no path back. Return a capped backoff instead, so the connection heals on its own once Redis returns. Extracted to a named export above the c8 ignore block so it can be tested
getAllBreachesFromDb swallowed a Redis fault and returned []. getBreaches() reads that as "the breaches table is empty" and answers by fetching the whole catalogue from HIBP and re-upserting it, per request.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
References:
Jira: MNTOR-5319
Incident doc: 2026-09-03 Monitor Website Unresponsive
Description
redisClient()declared a module-levelsingletonbut never read it, so every call opened a new ioredis connection and abandoned the previous one.getAllBreachesFromDb()calls it per request, so the web tier leaked roughly one socket per request until Memorystore hit its 64,998maxclientsceiling.Deploys and pod restarts have been silently protecting prod from this, as the connections would be cleared out.
Testing Before and After
Built for production twice, once on
mainand once on this branch, pointedREDIS_URLat a throwawaydocker run -d --name redis-demo -p 6379:6379 redis:7-alpine, then fired 300 requests at/en/breacheswith concurrency 10 while samplingredis-cli info clientsonce a second.Redis connections:
One socket per request, and they are never released. That last row is the incident in miniature: nothing reclaims them until the pod dies.
Response times, same 300 requests. Sharing one connection instead of 300 was the thing I was most worried about, and it turns out to be slightly faster, because removing a TCP handshake per request more than pays for the serialised reply.
Separately, stopping Redis for 8 seconds under a live client:
maindies at t≈1.2s withRedis Could not connect after 4 attempts.How to test
Each regression case fails on
main:client.test.tsreports 100 connections for 100 calls,util.test.tsthrows atretryStrategy(4), andhibp.test.tsgets[]instead of the Postgres rows.Checklist (Definition of Done)
docs/architecture/flows/breach-read-path.mdnow distinguishes a cache miss from a cache fault, andretryStrategycarries a comment saying why it must never throw.main.JSON.parse, which is out of scope here.Follow Up Action
Watch
redis.googleapis.com/clients/connectedfor 24 to 48h after deploy. It should stay near its ~450 floor instead of climbing.