Handling Backlogged RunRequests Triggered by Broken/Stuck Sensors

[dennisgera]

Hi team,

Is there a way to configure a sensor to either disregard outdated requests or a method to cancel accumulated RunRequests programmatically (e.g., script)?

Context

I had a sensor stuck in an "Evaluating" state for an extended period (two days). Once fixed, the system generated a RunRequest for every missed sensor tick during the downtime. This resulted in a sudden surge of runs being executed.

Questions:

1. Is there a feature allowing the automatic cancellation of RunRequests deemed too old?
2. Alternatively, is there a set of commands (script) to clear the queue of these backlogged RunRequests selectively?

Thanks!

[alangenfeld]

Is there a feature allowing the automatic cancellation of RunRequests deemed too old?

The current recommended approach would

1. be to use a cursor with your sensor
2. mutate that cursor in such a way to progress it forward.

https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors#idempotence-and-cursors

Alternatively, is there a set of commands (script) to clear the queue of these backlogged RunRequests selectively?

#14203

Also, recent versions of the UI include a Terminate all... button on the Queued runs tab.