fix(queue): retry queued requests on transient errors instead of dropping them

[TimeToBuildBob]

This is a carry of #115 (Erik's work) rebased onto current master, resolving the single conflict where #118 added persistqueue_path to RequestQueue.__init__ at the same site where #115 adds RETRY_STATUS_CODES. Resolution keeps both:

RETRY_STATUS_CODES = {429, 500, 502, 503, 504}

def __init__(self, client: ActivityWatchClient, persistqueue_path: Optional[str] = None) -> None:

All commits are authored by @ErikBjare. Erik: feel free to close this in favour of rebasing #115 yourself — this carry exists only to unblock #127 and #128 which stack on this base.

Why carry instead of waiting on #115: The conflict has been unresolved for 4 days since I noted it on #115 (#115 comment 2026-10-02). A fork carry lets #127/#128 land without further delay while keeping the option to supersede this PR with a direct rebase.

Closes #115 (if merged; Erik may prefer to merge #115 directly instead).

Stack:

• This PR (carry of fix(queue): retry queued requests on transient errors instead of dropping them #115): retry transient errors
• feat(queue): merge queued heartbeats per bucket before dispatch #127: pre-merge queued heartbeats per bucket before dispatch
• feat(queue): bulk-insert unmergeable drained events instead of one heartbeat each #128: bulk-insert unmergeable drained events

Fixes #32 (partially — #127 handles the merge half; #128 handles batch insert).

Problem

On master, RequestQueue permanently drops any queued heartbeat that gets an HTTP error, including 500 and 503. The retry-on-500 branch can never run: if e.response and ... is always false, because requests.Response.__bool__ returns False for every non-2xx status. A ConnectionError raised mid-dispatch (server died after connect) is also dropped, since only ConnectTimeout was caught. aw-server sends 503 when the heartbeat lock times out, so a briefly busy server silently loses watcher data.

Reproduce

python3 -c "import requests; r = requests.Response(); r.status_code = 503; print(bool(r))"   # False

Run the new tests against master's client.py:

git checkout origin/master -- aw_client/client.py
pytest -q tests/test_requestqueue.py   # 7 failed, 5 passed
git checkout HEAD -- aw_client/client.py
pytest -q tests/test_requestqueue.py   # 12 passed

Six of the seven failures are behavioural: each retryable status (429/500/502/503/504) and a connection error leave the queue empty. The seventh is an ImportError for the new _retry_delay.

What the tests cover

• 429/500/502/503/504: the request stays queued, then is delivered once the server recovers.
• 400: the request is dropped so a bad payload can't block the queue.
• ConnectionError: the request stays queued and the queue goes back to reconnecting.
• Retry-After: delta-seconds are honored, capped at 60s and floored at 0.5s; the HTTP-date form falls back to the default.

[greptile-apps]

Confidence Score: 5/5

[Medium risk] Changes how the request queue handles server errors.

The PR appears safe to merge.

What we checked:

• Dispatch tests use separate queues: _fresh_queue no longer drains the shared queue. Each caller supplies tmp_path, which keeps its queued requests separate.

Summary

Queued requests now survive connection failures, timeouts, and HTTP 429/500/502/503/504 responses. Retries use a bounded delay and honor numeric Retry-After values.

• Both previous findings are fixed: the new dispatch tests use separate queues, and dropped requests keep their exception details.
• No new actionable issues were found in the changes since the previous review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Read queued request] --> B[Send request]
  B --> C{Result}
  C -->|Success| D[Remove request]
  C -->|Connection failure or timeout| E[Keep request and reconnect]
  C -->|HTTP 429/500/502/503/504| F[Keep request and wait before retry]
  C -->|Other failure| G[Log exception and remove request]

Loading

_{Reviews (2) · Last reviewed commit: "fix(queue): isolate retry tests' queue, ..."}

[TimeToBuildBob]

@greptileai review

[TimeToBuildBob]

Both review findings are fixed in 82a7736 and Greptile now reports 5/5 with no new actionable issues. Seven checks have passed; the macOS test is still queued. No further code change or review round is needed. Merge remains with a maintainer (Bob has pull-only access); once CI finishes, this carry is ready for your review/merge or superseding with a direct rebase of #115.

[TimeToBuildBob]

Rebased onto current master after #124 and #125 landed.

Conflict was only in tests/test_requestqueue.py (both sides appended tests). Resolution keeps master's wait_for_queue_empty / disk-full coverage and this PR's retry tests. aw_client/client.py auto-merged: transient retry + Retry-After, bool add_request, and wait_for_queue_empty are all present.

Local: poetry run pytest tests/test_requestqueue.py — 18 passed. New head is 65148ab. Merge still needs a maintainer (pull-only); #115 can still supersede this carry.