Set RCU/WCU for PROVISIONED tables while maintaining support for autoscaling #178
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
#158
Description of changes:
Currently we do not configure DynamoDBConstants.READ_THROUGHPUT and DynamoDBConstants.WRITE_THROUGHPUT for PROVISIONED tables in our DynamoDB InputFormat. The reason for this is because PROVISIONED tables can potentially have auto-scaling enabled on Read and Write capacity and every time a new task starts we want to fetch from DDB the current capacity to make sure we are fully utilizing it. However we also use read and write throughput variables to calculate number of mappers so not setting this property initially will cause only one map task to be launched
These changes make it so that even for provisioned tables we initially set the read throughput and write throughput so that we can still estimate number of mappers that will be needed. However instead we pass a new configuration for PROVISIONED tables that indicated that throughput should be fetched every time a new task starts to account for auto-scaling.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.