[RFC] DynamoDB table

[mnapoli]

The goal of this discussion is to get feedback on a "DynamoDB table" construct.

Use case

Setting up DynamoDB tables in serverless applications.

Quick start

# serverless.yml
# [...]

constructs:
    db:
        type: dynamodb
        partitionKey:
            name: id
            type: string

How it works

On serverless deploy, the dynamodb construct will create a DynamoDB table preconfigured for production.

The table will be configured to scale on-demand (not provisioned). Costs will start at $0/month and will be proportional to the usage of the table.

Point in time recovery will be enabled to get continuous backups for 35 days.

Variables

All dynamodb constructs expose the following variables:

• tableName: the name of the DynamoDB table
• tableArn: the ARN of the DynamoDB table

These can be used to reference the queue from Lambda functions, for example:

constructs:
    db:
        type: dynamodb
        # ...

functions:
    my-function:
        handler: function.handler
        environment:
            TABLE_NAME: ${construct:db.tableName}

How it works: the ${construct:db.tableName} variable will automatically be replaced with a CloudFormation reference to the DynamoDB table.

Permissions

By default, all the Lambda functions deployed in the same serverless.yml file will be allowed full access to the DynamoDB table.

Configuration reference

Primary key

DynamoDB primary keys are either:

• partition key
• partition key + sort key (composite key)

When only using a partition key:

constructs:
    db:
        type: dynamodb
        partitionKey:
            name: id
            # The attribute type can be "string" (or "S"), "number" (or "N"), "binary" (or "B")
            type: string

When using a partition key + a sort key:

constructs:
    db:
        type: dynamodb
        partitionKey:
            name: projectId
            type: string
        sortKey:
            name: userId
            type: string

---

Would a construct like this help in your projects? What would be missing?

[t-richard]

Seems to be a regular use case for serverless applications.

On-demand seems a good default and I don't see a need for it to be configurable in a first version.

Point in time recovery is also a nice thing to have by default. But I think it makes sense to make it configurable because PITR is not included in the always free tier of dynamodb and and isn't necessary if you are using dynamodb as a cache store.

[mnapoli]

Ah good point, cache usage is a different use case to keep in mind.

Either "cache" is too different from "database" as a use case (in terms of options/default settings) and we need a separate construct, either we can use one for both. If it's just a matter of disabling PITR then that could be an option indeed.

[fredericbarthelet]

Thanks for putting together this proposal @mnapoli :)

The equivalent CloudFormation configuration of your proposed API is (excluding point in time recovery):

AWSTemplateFormatVersion: "2010-09-09"
Resources: 
  myDB: 
    Type: AWS::DynamoDB::Table
    Properties: 
      AttributeDefinitions: 
        - 
          AttributeName: "projectId"
          AttributeType: "S"
        - 
          AttributeName: "userId"
          AttributeType: "S"
      KeySchema: 
        - 
          AttributeName: "projectId"
          KeyType: "HASH"
        - 
          AttributeName: "userId"
          KeyType: "RANGE"
      BillingMode: PAY_PER_REQUEST

I fill like there is not much addition provided by Lift on top of using straight CloudFormation.

Here are a few additional recommendations I'd stick by to ensure users can easily bootstrap a one-table configuration with minimal DynamoDB knowledges upfront:

• naming primary key column with non-generic names is a one-table design code smell, I would not offer the possibility to actually name them and rather stick with PK and SK with string type.
• LSI does not incur additional cost when no filled with data, however, adding those indexes at a later stage of development of the application requires it to be re-created. I'd include by default the maximum number of 5 LSI from the ground up with generic names LSI1, LSI2, ... By default, I'd use the projection ALL type to project all attributes and the string type for those attributes.
• I'd add a GSIQuantity property, by default equal to 0, provisioning the corresponding number of GSI, following a GSI1_PK/ GSI1_SK couple pattern as they are added/removed
• I would enable by default TimeToLiveSpecification and StreamSpecification as they don't affect pricing and are therefore available when required

Using single-table design prevents user from actually leveraging this construct when:

• not knowing in advance data access patterns
• using GraphQL with AppSync

However, I feel like those use-cases only represents a small part of DynamoDB users. WDYT ?

[ThomasAribart]

@fredericbarthelet There are some use cases in which you'll want to use the same attribute (including PK/SK) accross several secondary indexes. The index attributes you suggest would be good by default, but I'd suggest you still offer the possibility to override them.

Beware that after the creation of the table, you can only add one GSI at a time. Also PITR does not allow you to return the db to the state it was 30 days ago, but rather creates a new db with the same state. I'd add those infos in the doc: You can restore the table to a previous state with the help of S3 and DataPipeline.

[LouisPinsard]

@fredericbarthelet
Concerning stream, it may be a good idea to let the user decide between DynamoDB stream and Kinesis stream.
Main differences from what I understand are:

• You will be charged for each item put in your Kinesis stream
• You can have as much lambda as you need triggered by your Kinesis stream VS limit of 2 lambda for DynamoDB stream.

[t-richard]

Maybe there is room for several components each targetted at a single use case ?

I'm not super familiar with DynamoDB and I'm using it mostly as key-value store for cache or simple use cases so what @mnapoli originally offered would be a good fit for those.

Other components could configure the table for their specific use case and create additional resources that may be needed.

[mnapoli]

Amazing, thanks all for the feedback! This RFC is definitely tackling too much at once, but at least it gets the ball rolling.

Do we have 3 different use cases here?

• cache: key-value store without durability. Dynamodb instead of redis lets us avoid VPCs, simpler to host, autoscales, cheap, etc.
• key-value store: (almost same as cache, without durability) no need for secondary indexes, LSI and GSI.
• database with single-table design: optimized with all recommendations mentioned above (e.g. you don't choose the PK/SK names, etc.)

These could be 3 different constructs.

WDYT?