`TimeoutError` on publish during a server restart

[superlevure]

Observed behavior

During a server rollout, a client that publishes to a stream can face a errors.TimeoutError:

Traceback (most recent call last):
  File "/workspace/app/nats_test.py", line 102, in publish_message
    await nats_client.publish(
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nats/js/client.py", line 201, in publish
    msg = await self._nc.request(
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nats/aio/client.py", line 1061, in request
    msg = await self._request_new_style(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nats/aio/client.py", line 1105, in _request_new_style
    raise errors.TimeoutError

Expected behavior

No Exception should be raised, the publish operation should be robust to a server rollout (like the subscribe operation).

Server and client version

• Server: 2.11.8
• Client: nats-py==2.11.0

Host environment

Server running in clustered mode with 3 nodes. The stream has 3 replicas.

Steps to reproduce

• Deploy NATS with Jetstream enabled on kubernetes using the official Helm Chart (nats-1.3.13) and 3 replicas.
• Create a test R3 stream
• Run the following script:

import asyncio

import nats

NATS_SERVER_URL = "nats://nats.nats:4222"

async def main() -> None:
    nats_client = await nats.connect(
        servers=NATS_SERVER_URL,
    )
    jetstream = nats_client.jetstream()

    count: int = 0
    while True:
        subject = "test"
        message = b"Hello, World! %d" % count
        await jetstream.publish(
            subject=subject,
            payload=message,
        )
        print(
            f"Published {message} to {subject}.",
        )
        await asyncio.sleep(0.1)
        count += 1


if __name__ == "__main__":
    asyncio.run(main())

• Perform a rollout of the server: kubectl rollout restart --namespace nats statefulset/nats
• After a few attempts, you should experience the following error on the client side:

nats: encountered error
nats.errors.UnexpectedEOF: nats: unexpected EOF
Traceback (most recent call last):
  File "/workspace/app/minimal_test.py", line 19, in main
    await jetstream.publish(
  File "/usr/local/lib/python3.11/site-packages/nats/js/client.py", line 201, in publish
    msg = await self._nc.request(
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nats/aio/client.py", line 1061, in request
    msg = await self._request_new_style(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nats/aio/client.py", line 1105, in _request_new_style
    raise errors.TimeoutError
nats.errors.TimeoutError: nats: timeout

[caspervonb]

Sounds like https://github.com/nats-io/nats-architecture-and-design/blob/main/adr/ADR-22.md would resolve this?

[superlevure]

Interesting, thank you for the link. We ended up implementing a retry mechanism similar to ADR-22 and it did solve the issue for us.

One thing to note though, it needs to be associated with message deduplication because sometimes nats.errors.TimeoutError is thrown but the message reaches the server anyway, without setting Nats-Msg-Id it results in duplicates.

[Jarema]

We can add an opt-in retry, as as you see - it's easy to end up with a duplicate in the stream.

[caspervonb]

ADR-22 only targets NoResponders tho, is there precedent for retrying on timeout in the other clients @Jarema?

[caspervonb]

#734 implements ADR-22.