Skip to content

fix: recover zombie connections on KeepAliveTimeout - #160

Open
FlavioPulli wants to merge 1 commit into
evolution-foundation:developfrom
FlavioPulli:fix/keepalive-timeout-handling-develop
Open

fix: recover zombie connections on KeepAliveTimeout#160
FlavioPulli wants to merge 1 commit into
evolution-foundation:developfrom
FlavioPulli:fix/keepalive-timeout-handling-develop

Conversation

@FlavioPulli

@FlavioPulli FlavioPulli commented Aug 4, 2026

Copy link
Copy Markdown

With EnableAutoReconnect disabled (whatsmeow.go:464), a TCP connection that dies silently (no FIN/RST from the server — NAT timeout, network flap) never emits events.Disconnected: keepalive pings just keep timing out and the instance stays flagged as connected while being unable to send or receive. Only a manual restart fixes it. whatsmeow's own docs on KeepAliveTimeout suggest exactly this: "Clients may use this event to decide to force a disconnect+reconnect faster."

This handles KeepAliveTimeout and KeepAliveRestored:

  • both are forwarded to the CONNECTION webhook subscription so operators can observe socket health;
  • after 3 consecutive timeouts the instance is restarted through the same non-blocking ReconnectClient path the Disconnected handler already uses (exact-match on the count so an ongoing restart isn't duplicated).

We run multiple production instances behind this and it eliminated our "zombie instance" incidents (instance reported connected, no traffic flowing, external watchdog required).


Retargeted to develop (replaces #126), as requested by @iagocotta in #128 (comment).

Note that main and develop have no common ancestor — the GitHub API refuses to compare them (No common ancestor between main and develop) — so the base branch of the original PR could not simply be edited. This branch was created from develop and the commit cherry-picked onto it.

Verified on this branch: go build ./... and go vet ./... both pass.

Summary by Sourcery

Handle keepalive timeout and restoration events to recover zombie connections and expose socket health via connection webhooks.

Bug Fixes:

  • Restart instances after three consecutive keepalive timeouts to recover silently dead TCP connections when auto-reconnect is disabled.

Enhancements:

  • Forward KeepAliveTimeout and KeepAliveRestored events to CONNECTION webhooks and global queues for better observability of connection health.

With EnableAutoReconnect disabled, a TCP connection that dies silently
(no FIN/RST from the server) never emits events.Disconnected: keepalive
pings just keep timing out and the instance stays flagged as connected
while being unable to send or receive — a zombie that only a manual
restart fixes. whatsmeow's docs explicitly suggest using KeepAliveTimeout
to force a faster disconnect+reconnect.

Handle KeepAliveTimeout and KeepAliveRestored: both are forwarded to the
CONNECTION webhook subscription so operators can observe socket health,
and after 3 consecutive timeouts the instance is restarted through the
same non-blocking ReconnectClient path the Disconnected handler already
uses (exact-match on the count so ongoing restarts aren't duplicated).
@sourcery-ai

sourcery-ai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Reviewer's Guide

Handle whatsmeow keepalive timeout/restored events to surface socket health via CONNECTION webhooks and automatically restart zombie instances after repeated timeouts, while wiring the new events into global queue routing.

Sequence diagram for keepalive timeout handling and zombie connection recovery

sequenceDiagram
    participant WhatsmeowClient
    participant MyClient
    participant whatsmeowService

    WhatsmeowClient->>MyClient: events.KeepAliveTimeout
    MyClient->>MyClient: myEventHandler
    MyClient->>whatsmeowService: CallWebhook (eventType KeepAliveTimeout)
    whatsmeowService->>whatsmeowService: sendToQueueOrWebhook (subscription CONNECTION)
    alt [ErrorCount == 3]
        MyClient->>whatsmeowService: ReconnectClient(instanceID)
    end
    WhatsmeowClient->>MyClient: events.KeepAliveRestored
    MyClient->>MyClient: myEventHandler
    MyClient->>whatsmeowService: CallWebhook (eventType KeepAliveRestored)
    whatsmeowService->>whatsmeowService: sendToQueueOrWebhook (subscription CONNECTION)
Loading

Flow diagram for routing keepalive events to CONNECTION global queues

flowchart LR
    A[Receive eventType] --> B{eventType in
Connected / PairSuccess /
TemporaryBan / LoggedOut /
ConnectFailure / Disconnected /
KeepAliveTimeout / KeepAliveRestored}
    B -->|yes| C[Set globalEventType = CONNECTION]
    C --> D[SendToGlobalQueues]
    B -->|no| E[Route using other mappings]
Loading

File-Level Changes

Change Details Files
Handle KeepAliveTimeout by forwarding it to webhooks/global queues and triggering a non-blocking client reconnect after 3 consecutive timeouts to recover zombie connections when auto-reconnect is disabled.
  • Add a KeepAliveTimeout case in MyClient.myEventHandler that marks doWebhook true, populates postMap, and logs keepalive timeout details with error count and last success timestamp.
  • Implement an exact-match ErrorCount == 3 guard that spawns a goroutine to call service.ReconnectClient using the same restart path and logging as the Disconnected handler, avoiding duplicate restarts during ongoing recovery.
pkg/whatsmeow/service/whatsmeow.go
Handle KeepAliveRestored by forwarding it to webhooks/global queues and logging restoration of keepalive pings for connection observability.
  • Add a KeepAliveRestored case in MyClient.myEventHandler that marks doWebhook true, sets the event name in postMap, and logs keepalive restoration.
  • Extend CallWebhook routing to treat KeepAliveTimeout and KeepAliveRestored as CONNECTION events when the CONNECTION subscription is present.
  • Extend SendToGlobalQueues routing (both overloads) to map KeepAliveTimeout and KeepAliveRestored into the CONNECTION globalEventType.
pkg/whatsmeow/service/whatsmeow.go

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • The KeepAliveTimeout logic hardcodes the restart threshold at exactly 3 timeouts; consider making this threshold configurable (or at least a named constant) so operators can tune it without code changes.
  • In the KeepAliveRestored case you don't populate postMap["data"] as you do for KeepAliveTimeout; if webhooks expect a consistent structure, consider including relevant metadata (e.g., LastSuccess or a timestamp) or explicitly documenting the difference.
  • Within the KeepAliveTimeout handler you call loggerWrapper.GetLogger multiple times with the same instanceID; consider storing the logger in a local variable to avoid repeated lookups and keep the logging code a bit cleaner.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- The `KeepAliveTimeout` logic hardcodes the restart threshold at exactly 3 timeouts; consider making this threshold configurable (or at least a named constant) so operators can tune it without code changes.
- In the `KeepAliveRestored` case you don't populate `postMap["data"]` as you do for `KeepAliveTimeout`; if webhooks expect a consistent structure, consider including relevant metadata (e.g., `LastSuccess` or a timestamp) or explicitly documenting the difference.
- Within the `KeepAliveTimeout` handler you call `loggerWrapper.GetLogger` multiple times with the same `instanceID`; consider storing the logger in a local variable to avoid repeated lookups and keep the logging code a bit cleaner.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant