Helped with a SQS event processor

[Iamrodos]

We have an SQS-triggered Lambda that processes events, where each event can have multiple handlers (send email, update records, trigger downstream events, etc.). Before durable functions, if handler 3 of 5 failed, the SQS retry would reprocess handlers 1-2 unnecessarily. These handlers were not always idempotent, causing duplicate side effects. Its been a TODO to resolve and durable functions turned into an easy way to do this.

We wrapped our event processor in @durable_execution and use ctx.step() to checkpoint:

1. Event creation - The initial write to DynamoDB
2. Each handler - Individual steps so completed handlers are skipped on replay

for i, handler in enumerate(handlers):
    if ctx:  # Durable mode - wrap each handler in a step
        ctx.step(lambda _: handler(event), name=f"handler-{i}-{handler.__name__}")
    else:  # Standard mode
        handler(event)

We use a DURABLE_EXECUTION_ENABLED env var to switch between durable and standard BatchProcessor paths. This lets us roll back instantly without code changes if issues arise.

We include the handler index in step names (handler-0-send_email) to avoid collisions when multiple handlers share the same __name__.

The only issue we hit was a circular dependency on CloudFromation when trying to write the role for the Lambda. The IAM role needs the Lambda ARN for checkpoint permissions, but the Lambda needs the Role ARN. Would like to find a way to solve that.

[yaythomas]

hi @Iamrodos, this sounds like a nifty solution! 🎉

Please check here for a sample using CloudFormation that defines a Role and references it in the DurableFunction resource with Role: !GetAtt DurableFunctionRole.Arn:

https://docs.aws.amazon.com/lambda/latest/dg/durable-getting-started-iac.html#durable-iac-cloudformation

Other than CloudFormation, I can't tell what type of iterable handlers is from your sample, but please double-check that it's something like list that enumerates deterministically (i.e it loops in the same order each time).

And also, watch our for the 15 minute execution limit as it interacts with automatic step retries, defined here. The failing step will attempt retrying automatically.

Some suggestions on ESM execution error handling: https://docs.aws.amazon.com/lambda/latest/dg/durable-invoking-esm.html#durable-esm-error-handling

If your handlers are not serial, you could also consider using parallel to run concurrently.