CeleryIntegration with Redbeat scheduler seems to send 2 In Progress Check ins leading 1 to Timeout

[Fwang36]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.28.0

Steps to Reproduce

Helping a user in this Zendesk ticket (last several responses): https://sentry.zendesk.com/agent/tickets/152175

It seems like the celery integration used with the Redbeat scheduler is creating two in-progress check in events at the scheduled time. The worker only sends 1 OK check-in event leading to the other in-progress event to timeout. I was able to reproduce this with a task that runs every minute.

Using the celery beat scheduler instead of redbeat works as expected.

My sentry settings -

@signals.beat_init.connect
@signals.celeryd_init.connect
def init_sentry(**_kwargs):
    sentry_sdk.init(
        dsn="<DSN>",
        send_default_pii=True,
        integrations=[CeleryIntegration(
            monitor_beat_tasks=True,
        )],
        debug=True,
    )

Run commands -

celery -A tasks worker --loglevel=INFO

celery -A tasks beat --scheduler redbeat.RedBeatScheduler --loglevel=INFO

When the task starts, I get 2 monitor check ins per every time the task is run (task runs 2 times in screenshot) in beat terminal -

In the worker terminal, I get 1 check in per task (as expected) -

In the UI, it looks like this, where the In Progress check ins will then timeout after the set threshold -

Link to my monitor

Expected Result

Sends only 1 in progress event when the task begins.

Expect behavior to match celery beat scheduler

Actual Result

Sends 2 in progress events when the task begins.

[antonpirker]

Hey @Fwang36 thanks for the great issue description! I will have a look

[antonpirker]

I have now reproduced your error:

• The beat task sends a in_progress for my task with the checkin-id a.
• Then the celery process sends a in_progress for my task with the checkin-id b,
• then a ok checkin with checkin-id b,
• and finally a ok checkin with checkin-id a.

That is the project I used: https://github.com/antonpirker/testing-sentry/tree/main/test-celery-redbeat

So clearly something is off. I prioritize this fix. I'll keep you posted, sometime next we should have this fixed.

[antonpirker]

Here my beat log:

Here my celery worker log:

[antonpirker]

I have now tested by disabling celery-redbeat in my test project, using the default celery scheduler (celery.beat.PersistentScheduler) and I get the same result. So this issue is affecting more people then just redbeat users.

[sentrivana]

This looks like possible double instrumentation to me.

Is it possible you're using both monitor_beat_tasks=True and wrapping the tasks in the @sentry_sdk.monitor decorator? They are mutually exclusively per my understanding -- i.e., you either let Sentry monitor all beat tasks automagically, or you let it monitor none and you hand-pick whichever you want with the decorator.

[sentrivana]

Nevermind, I get the extra in_progress check-in too. Looking into this.

[sentrivana]

As far as I can tell, the problem is that we're patching redbeat's maybe_due and sending an in-progress check-in there, even though maybe_due might not end up actually scheduling the task if it determines it's not supposed to run. Fix here: #4395

We can cut a release with this next week.

[antonpirker]

Small related docs update: getsentry/sentry-docs#13751

[sentrivana]

The fix for this is now out in 2.29.0!