SQS Integration

[snlamm]

SQS

tldr:

• AWS SQS integration
• Create/update SQS queues via the config
• Supports the same outward facing api and error handling as the other integrations

Details:

Fetching Urls

SQS requires a queue url for most operations - it usually doens't accept just the name. The queue url can either be hardcoded into the app code or fetched via the sdk api. If the queue has not been created yet, however, then it needs to be created before the url can be fetched.

It's not a good idea to get this url for the first time when performing an in-app operation, such as the first time a job is dispatched. First, it requires time to fetch the url and even more time to create/update the queue if needed. Second, if multiple dispatches occur asynchronously, multiple unecessary api requests are sent to aws, which might also cause errors.

The solution is to get the urls and/or create/update the queue on app-startup. The queue names and urls will be specified in the app config.
For example:

		"connections": {
			"sqs-connection": {
				"driver": "sqs",
				"queues": {
					"test-queue-name": { ...queue options here },
					"test-queue-name2": { "QueueUrl": "https://sqs.us-east-1..." }
				}
			}
		}
        

It works as follows:

• On app start, grind-queue calls AWS to create/update your specified queues as necessary and grabs the URLs.

• For any queue options not specified in the config, we'll use the AWS defaults.

• The queue name default is reserved for setting queue defaults different from AWS's.

• If you want to use an SQS queue managed manually without grind-queue, set the QueueUrl option.

• Grind-queue will use that url and, on startup, will skip the inital calls to AWS.

This fetching required me to promisify some of the internal logic of getting queues in Queue.js, but that doesn't effect any functionality of the other services or impact outward usage.

Concurrency

SQS can batch recieve and delete messages up to 10 at a time. Concurrency takes advantage of this. If I set concurrency to 2, only one listener is used but a concurrency of 2 is set for it, so up to 2 jobs are received at a time. If I set concurrency to 13, then two listeners are used, one with a concurrency of 10 and the other with 3.

Deleting Jobs

When a job is received from an SQS queue, it is not automatically deleted. After a delay (default 30 seconds), the job becomes visible to the queue and will be delivered again, causing duplicate processing. Any delay in processing, therefore, can cause duplicates.

To avoid duplicates, jobs are deleted from SQS the moment they are received. Retries and retryDelays are handled on the app level by re-running the job handler with the original job data-payload.

[shnhrrsn]

This look great! Can you update to add tests? The other drivers recently had them added, so just follow that pattern using a docker image that emulates the SQS API, it looks like there's a bunch of them.

[snlamm]

Updated and tests passing. Used this docker image: https://github.com/vsouza/docker-SQS-local. Rebased this branch to include the most recent changes to the target branch.