This is a sample project to use map state in a state machine to loop through a list of events and invoke a Lambda to process tham and store the erros events in a SQS queue. In the subsequest run the Lambda receives the error events from the SQS to retry the processing.
This sample state machine consists of two Lambdas, one SQS, one SNS and CloudWatch alarms for Lambda. The first Lambda generates some sample data with sequence number, random strings and generated Unix time stamp. The map state loops through the list generated by the first Lambda and processes them by inserting the records into a DynamoDB table. If any insertion fails then they are collected in a list and pushed into a SQS queue and the state machine fails. If all the generated data gets processed successfully the state machine ends in a success state. In the subsequent run of the state machine, the first lambda pulls the error events from the SQS (if any) and reprocesses them. If for any reason the processing fails even after third retry of the process, they are inserted into a dead letter queue.The entire stack is created using CloudFormation.
- Create a Customer Managed KMS Key in the region where you want to create the stack.
- Modify the KMS Key Policy to let the IAM user encrypt / decrypt using any resource using the created KMS Key.
- Clone the repository.
- Create a S3 bucket to store the CFT and python code zip file.
- Create the folders - carnation/cft/nested-stacks, carnation/code/state-machine, carnation/code/python
- Upload the following YAML templates to carnation/cft/nested-stacks/
- cloudwatch-stack.yaml
- dynamodb-stack.yaml
- lambda-function-stack.yaml
- sqs-stack.yaml
- iam-role-stack.yaml
- sns-stack.yaml
- Upload the following YAML templates to carnation/cft/
- carnation-root-stack.yaml
- Upload state machine ASL to the folder cft/state-machine/
- Zip and Upload the Python files to carnation/code/python
- Create the entire using by using the root stack template carnation-root-stack.yaml by providing the required parameters.
- To execute the state machine use the following input:
{
"start_seq": 1,
"end_seq": <integer>
}
The above input will create a payload consisting of (end_seq - start_seq) elements.
In the subsequent run use the following input to just reprocess the events from the SQS queue:
{
"start_seq": 0,
"end_seq": 0
}
Post message in my blog (https://blog.subhamay.com)
Contributors names and contact info
Subhamay Bhattacharyya - [email protected]
- 0.1
- Initial Release
This project is licensed under Subhamay Bhattacharyya. All Rights Reserved.
- AWS [Dynamically process data with a Map state] (https://docs.aws.amazon.com/step-functions/latest/dg/sample-map-state.html)