Broken connections on Celery worker with logging enabled

[mcboman]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.29.1

Steps to Reproduce

Hi,

I'm running a Celery worker with 16 concurrent workers and recently enabled the experimental logging feature in the Python SDK. Everything works fine initially, but after a while I start seeing this error repeatedly:

[2025-06-08 08:23:14,183: WARNING/ForkPoolWorker-27] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2570)')': /api/4509207728947280/envelope/

This happens under high task load, and workers eventually become unresponsive. If I disable the Sentry logging integration, the problem disappears.

Setup details:

Celery 5.5.2

Python Sentry SDK versions tried: 2.28.0 and 2.29.1

Python version: 3.12.8

Worker command:

celery worker \
  --concurrency=16 \
  --loglevel=info \
  -Q q1,q2 \
  --max-memory-per-child=2621440 \
  --max-tasks-per-child=10 \
  --without-gossip \
  --without-mingle \
  --heartbeat-interval=10

Any idea what could be causing this SSL error with the logging feature?

Expected Result

No SSL errors

Actual Result

[2025-06-08 08:23:14,183: WARNING/ForkPoolWorker-27] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2570)')': /api/4509207728947280/envelope/

[sentrivana]

Hey @mcboman, thanks for raising this.

To clarify, when you say

If I disable the Sentry logging integration, the problem disappears.

Do you mean you add LoggingIntegration to disabled_integrations in your init, or that you turn off/remove the experimental enable_logs option?

Before the SSL errors start popping up, is there any other error visible in the logs? I'm not sure where the SSL errors could be coming from -- my hunch is that it's a consequence of something else happening.

How many logs approximately are your workers sending to Sentry? If you adjust sentry_logs_level to a higher level (see below), does it change anything?

import logging
from sentry_sdk.integrations.logging import LoggingIntegration

sentry_sdk.init(
    ...
    integrations=[LoggingIntegration(sentry_logs_level=logging.ERROR),]  # or maybe logging.WARNING; default is logging.INFO
)

[mcboman]

Hi @sentrivana

It is the experimental feature for logging and not the instrumentation. I have attached the code for my sentry.init

Peak logs is 90k pr half hour.

sentry_sdk.init(
            dsn="<dsn>",
            send_default_pii=True,
            integrations=[
                CeleryIntegration(
                    monitor_beat_tasks=True,
                ),
                SqlalchemyIntegration(),
            ],
            traces_sample_rate=0.1,
            profile_session_sample_rate=0.1,
            profile_lifecycle="trace",
            _experiments={
                "enable_logs": True,
             },
        )