camunda cloud channel disconnect after a while

[EyalCMX]

we are working with
channel = create_camunda_cloud_channel(
client_id=WORKFLOW_ZEEBE_AUTH_CLIENT_ID,
client_secret=WORKFLOW_ZEEBE_AUTH_CLIENT_SECRET,
cluster_id=cluster_id
)

and we get lots of exceptions
"Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."
then after a while its stop but zeebe client dont get any task
its look like its not really connected but its not trying to reconnect any more

when we use
channel = create_insecure_channel(hostname=ZEEBE_GATEWAY_HOSTNAME, port=ZEEBE_GATEWAY_PORT)

we dont have this problem

we also have java client and in java we rarely see this exception and it always manages to connect at the end

anyone can point us to the problem?

example for logs we get(we get it much more)
{"time": "2022-07-12T11:40:47Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}
{"time": "2022-07-12T11:50:48Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}
{"time": "2022-07-12T11:51:38Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}
{"time": "2022-07-12T12:00:57Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}
{"time": "2022-07-12T12:01:07Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}
{"time": "2022-07-12T12:08:10Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}
{"time": "2022-07-12T12:14:42Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}
{"time": "2022-07-12T12:15:51Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}
{"time": "2022-07-12T12:19:21Z", "level": "WARNING", "pid": 1, "thread": 140004807083840, "loggerName": "pyzeebe.worker.job_poller", "message": "Failed to activate jobs from the gateway. Exception: ZeebeBackPressureError(). Retrying in 5 seconds..."}

[pleubner]

Bump ! Anyone have an answer ?

[EyalCMX]

look on this PR
#325