v1.8.3-1 noble stops after receiver.amplify.nginx.com timeout

[fersk]

We recently migrated a new server to 24.04 noble and are using agent v1.8.3-1~noble. Getting the error below when receiver.amplify.nginx.com times out. Agent stops.

2025-02-21 02:30:50,530 [1172] supervisor uncaught exception during run time
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 537, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 1428, in getresponse
    response.begin()
  File "/usr/lib/python3.12/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/socket.py", line 707, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/gevent/ssl.py", line 625, in recv_into
    return self.read(nbytes, buffer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/gevent/ssl.py", line 440, in read
    self._wait(self._read_event, timeout_exc=_SSLErrorReadTimeout)
  File "src/gevent/_hub_primitives.py", line 317, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 322, in gevent._gevent_c_hub_primitives.wait_on_socket
  File "src/gevent/_hub_primitives.py", line 313, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 314, in gevent._gevent_c_hub_primitives._primitive_wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 46, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_hub_primitives.py", line 55, in gevent._gevent_c_hub_primitives.WaitOperationsGreenlet.wait
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 845, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 472, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 791, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 539, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 371, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=5.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/amplify/agent/supervisor.py", line 356, in talk_to_cloud
    context.http_client.post('agent/', data=root_object)
  File "/usr/lib/python3/dist-packages/amplify/agent/common/util/http.py", line 128, in post
    return self.make_request(url, 'post', data=data, timeout=timeout, json=json)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/amplify/agent/common/util/http.py", line 110, in make_request
    raise e
  File "/usr/lib/python3/dist-packages/amplify/agent/common/util/http.py", line 94, in make_request
    r = self.session.post(
        ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=5.0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/amplify/agent/main.py", line 147, in run
    daemon_runner.do_action()
  File "/usr/lib/python3/dist-packages/amplify/agent/common/runner.py", line 42, in do_action
    self.app.run()
  File "/usr/lib/python3/dist-packages/amplify/agent/supervisor.py", line 277, in run
    self.talk_to_cloud(root_object=context.objects.root_object.definition)
  File "/usr/lib/python3/dist-packages/amplify/agent/supervisor.py", line 375, in talk_to_cloud
    self.cloud_talk_delay = exponential_delay(self.cloud_talk_fails)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/amplify/agent/common/util/backoff.py", line 35, in exponential_delay
    return randint(0, period_size - 1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/random.py", line 336, in randint
    return self.randrange(a, b+1)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/random.py", line 312, in randrange
    istop = _index(stop)
            ^^^^^^^^^^^^
TypeError: 'float' object cannot be interpreted as an integer
2025-02-21 02:30:56,092 [1172] supervisor failed POST "[https://receiver.amplify.nginx.com:443/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/](https://receiver.amplify.nginx.com/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/)", exception: "HTTPSConnectionPool(host='receiver.amplify.nginx.com', port=443): Read timed out. (read timeout=5.0)"
2025-02-21 02:30:56,092 [1172] supervisor [None] post [https://receiver.amplify.nginx.com:443/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/](https://receiver.amplify.nginx.com/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/) 500 795 0 5.547
2025-02-21 02:30:56,096 [1172] supervisor agent stopped, version=1.8.3-1 pid=1172 uuid=05e5b1d0f22e5658ad8c53582aa99057 

We have multiple other servers running 1.8.2-1~jammy on 22.04 jammy and the timeout is handled properly there:

2025-02-21 02:31:09,561 [42374] bridge_manager [bc35d14c1d8530a3eae40b5d885df9bd] post [https://receiver.amplify.nginx.com:443/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/](https://receiver.amplify.nginx.com/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/) 202 3814 0 0.421
2025-02-21 02:31:19,093 [42374] supervisor [e490f00104cfc3f422d00939cc35c9ac] post [https://receiver.amplify.nginx.com:443/1.4/9890f479ff7c57dce75cc2e09560e9f1/agent/](https://receiver.amplify.nginx.com/1.4/9890f479ff7c57dce75cc2e09560e9f1/agent/) 200 86 5 0.257
2025-02-21 02:32:14,600 [42374] bridge_manager failed POST "[https://receiver.amplify.nginx.com:443/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/](https://receiver.amplify.nginx.com/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/)", exception: "HTTPSConnectionPool(host='re>
2025-02-21 02:32:14,600 [42374] bridge_manager [None] post [https://receiver.amplify.nginx.com:443/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/](https://receiver.amplify.nginx.com/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/) 500 3817 0 5.008
2025-02-21 02:32:14,601 [42374] bridge_manager failed to push data due to ReadTimeout
2025-02-21 02:33:15,293 [42374] bridge_manager [194bcce98ba9c884c0f6eab09f090ff2] post [https://receiver.amplify.nginx.com:443/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/](https://receiver.amplify.nginx.com/1.4/9890f479ff7c57dce75cc2e09560e9f1/update/) 202 5144 0 0.657

[jaymoh]

I'm also experiencing this issue after upgrading one of my servers to Ubuntu 24. The timeout was handled fine in Ubuntu 20.04