A CD/CI tool for developing micro-services components using AWS Lambda function (python2.7) and managing other AWS resources.
Apex, Serverless, Zappa, sceptre
Most of the open-source apps are very much opinionated and the model they employ doesn't always fit for the actual individual use case. In fact, there aren't a lot of flexibility provided. In addition, abstraction makes it hard to diagnose issues.
The tool I have in mind should allow raw inputs, be atomic, has important functionalities like the others and easy to use to a certain degree (You need to know what you are doing. E.g. I don't expect you to use it for creating stack if you don't know how to write a raw CloudFormation). Hence born lmdo.
- Initialize project via Github boiler plate
- Use of CloudFormation templates in either json or yaml format
- Use of CloudFormation paramter files in either json or yaml format
- Manage one or more CloudFormation stacks
- Use of swagger template for API Gateway
- Manage API Gateway resources like deployments and stages
- Automatically generate API Gateway for Lambda functions
- Manage life cycles of AWS Lambda functions
- Offer two type of managed Lambda functions: Django wsgi wrapper and CloudWatch Event scheduler dispatcher
- Maintain Lambda function heart beat
- CloudWatch log output on CL
- Upload any files to S3 bucket
- Installation
- Project initiation
- Basic configuration
- One step deployment
- CloudFormation
- Lambda function
- API Gateway
- CloudWatch events
- CloudWatch logs
- S3 Upload
Installing via pypi:
$ sudo pip install lmdo
Installing via code (Recommended, as lmdo is under active development at the moment):
$ git pull https://github.com/MerlinTechnology/lmdo.git
$ cd lmdo
$ sudo pip install -U ./
Note: All lmdo commands need to be run at the same directory of the lmdo.yaml
file
To initiate your project, run:
$ lmdo init <project_name>
This will create you named project folder and the sample lmdo configuration file lmdo.yaml
.
If you already have an existing project, you can run:
$ lmdo init config
The configuration file lmdo.yaml
will be copied to your current directory. If there is already one, the new configuration file will be renamed to lmdo.yaml.copy
To start a project by using a github boiler plate, run:
$ lmdo bp fetch <url>
The repo will then be copied from github to your current project folder without all the git folders or files
-
AWS credentials
You can either use session (
Profile
) or configure AWS key and secret (Region, AWSKey, AWSSecret
) inlmdo.yaml
When using session, you will need to create two files:
~/.aws/config and ~/.aws/credentials
Details please ref to AWS CLI
Note: If explicitly using config options
Region, AWSKey, AWSSecret
, it's recommended to define them in the environment. Using syntax like$env|YOUR_ENV_VAR
lmdo will replace them with the actual environment value. -
Other mandatory configuration Options
Service
: The name of your service/projectUser
: The user that deploys the service/projectStage
: The deployment stage
To user a different configuration file (must be in yaml format), you can use command line option --config
, for example:
$ lmdo deploy --config=custom-config-file.yaml
To deploy your entire service, run:
$ lmdo deploy
To delete, run:
$ lmdo destroy
They will be replaced with correct value during deployment
$env|ENV_VAR_NAME
: Environment variables, can be used both in parameters and templates.
$template|[relative/path/from/repo/to/template/]template-file-name
: Nested stack template to be used to construct proper S3 bucket url for stack resource TemplateURL
, mostly used in templates.
$stack|stack-name::output-key
: The value of an existing stack's output based on key name. Can be used both in parameters and templates.
Note:
- the stack referring to must exist before deployment.
- Recommand to avoid use those variable in the template so to keep the template complying to AWS. Instead, use those variables in the parameter file to pass in as parameters
-
Single CloudFormation template without parameters
CloudFormation: Stacks: - Name: your-stack-name TemplatePath: relative/path/to/template
-
Single CloudFormation template with parameters. You can either provide a single file or a directory that contains all the parameter files. If a directory is provided, lmdo will combine all files into one during deployment.
CloudFormation: Stacks: - Name: your-stack-name TemplatePath: relative/path/to/template ParamsPath: relative/path/to/params/file/or/directory
-
CloudFormation using S3 bucket
CloudFormation: S3Bucket: your.bucket.url Stacks: - Name: your-stack-name TemplatePath: relative/path/to/template ParamsPath: relative/path/to/params/file/or/directory
-
Single CloudFormation template with nested stacks
CloudFormation: S3Bucket: your.bucket.url TemplateRepoPath: relative/path/to/nested/stack/template/directory Stacks: - Name: your-stack-name TemplatePath: relative/path/to/template ParamsPath: relative/path/to/params/file/or/directory
Note:
a. You must provide
S3Bucket
for nested stacks as it'll be used for uploading all the templates to.b. All nested stack templates must reside in
TemplateRepoPath
. If not given, lmdo will look for nested stack template (see point c below) from the project folder by default.c. Using syntax like
TemplateURL: $template|[relative/path/from/repo/to/template/]your-nested-stack-template-file-name
in your master template stack resource, lmdo will replace the syntax to appropriate S3 url.d. You can use
DisablePrefix
option to create stack with exact name you give -
Multiple CloudFormation Stacks
CloudFormation: S3Bucket: your.bucket.url TemplateRepoPath: relative/path/to/nested/stack/template/directory Stacks: - Name: your-stack-name-1 TemplatePath: relative/path/to/template-1 ParamsPath: relative/path/to/params/file-1/or/directory-1 - Name: your-stack-name-2 TemplatePath: relative/path/to/template-2 ParamsPath: relative/path/to/params/file-2/or/directory-2
Parameter file can be in either .json
or .yaml
format.
For json file, you can use two types of syntax:
-
Standard AWS stack parameter format
[ { "ParameterKey": "your-parameter-key-1", "ParameterValue": "your-parameter-value-1" }, { "ParameterKey": "your-parameter-key-2", "ParameterValue": "your-parameter-value-2" } ]
-
lmdo json format
{ "your-parameter-key-1": "your-parameter-value-1", "your-parameter-key-2": "your-parameter-value-2" }
-
For yaml file, the format as follow:
your-parameter-key-1: your-parameter-value-1 your-parameter-key-2: your-parameter-value-2
-
Available reserved utility variables
They will be replaced with correct value during deployment
$env|ENV_VAR_NAME
: Environment variables, can be used both in parameters and templates.$stack|stack-name::output-key
: The value of an existing stack's output based on key name. Can be used both in parameters and templates.$template|[relative/path/from/repo/to/template/]template-file-name
: Nested stack template to be used to construct proper S3 bucket url for stack resourceTemplateURL
, mostly used in templates.Note:
The stack referring to must exist before deployment.
For
CommaDelimitedList
type, you can do"$env|ENV_VAR_NAME1, $env|ENV_VAR_NAME2"
. Same to$stack|*
.
To create your CloudFormation, run:
$ lmdo cf create
To update or delete, run the similar command using update
or delete
keyword
To use change-set instead of directly update stack, use -c
or --change_set
option:
$ lmdo cf create -c
Stack event will be output by default, if you want to hide it, use -he
or --hide-event
option:
$ lmdo cf create -he
For only change one specific stack, use option --stack=
lmdo facilitates packaging, uploading and managing your lambda function. Out of box, it also provides support for two types of lambda function wrapper: wsgi and event dispatcher apart from the standard Lambda function.
VirtualEnv: False
Lambda:
- function-1 configuration
- function-2 configuration
...
Note:
- If you are using virtualenv, please set
VirtualEnv
toTrue
- The actual deployed function name created by lmdo will be using
<user>-<stage>-<service-name>-<FunctionName>
Type
: default default
. Other availabe types are wsgi
and cron_dispatcher
MemorySize
: default 128
Runtime
: default python2.7
(Note: lmdo only support python at the moment)
Timeout
: default 180
HeatUp
: default False
. Provide CPR for lambda function to keep container alive. Only avaiable for wsgi
and default
functions.
HeatRate
: default rate(4 minutes)
before container becomes inactive. Only avaiable for wsgi
and default
functions.
VPC configuration:
VpcConfig:
SecurityGroupIds:
- security-group-id-1
- security-group-id-2
SubnetIds:
- subnet-id-1
- subnet-id-2
Runtime environment variables
EnvironmentVariables:
MYSQL_HOST: host-url
MYSQL_PASSWORD: password
MYSQL_USERNAME: username
MYSQL_DATABASE: dbname
Role and policies:
RoleArn: your-role-arn
or
RolePolicy:
AssumeRoles:
- sns.amazonaws.com
PolicyDocument: file/path/to/your/policy
ManagedPolicyArns:
- your-managed-policy-arn
Note:
$region
and$accountId
are available to use in yourPolicyDocument
so you don't need to hard-code them- Only one of
RoleArn
andRolePolicy
required. - If both provided,
RolePolicy
takes over. - If none provided, lmdo will create a default role with default policy
- the default role will assume role of apigateway, lambda, events, ec2.
- the default policy will allow
lambda:InvokeFunction
,lambda:AddPermission
,lambda:RemovePermission
on the lambda function,log:CreateLogGroup
,logs:CreateLogStream
,logs:PutLogEvents
,ec2:DescribeNetworkInterfaces
,ec2:CreateNetworkInterface
andec2:DeleteNetworkInterface
actions - Only extra assume roles and policies need to be configured other than the default
-
Standard lambda function
Requirements:
The invokable lambda function files need to be placed on the top level of the project folder.
Put all your dependent packages in
requirements.txt
Configuration:
Lambda: - S3Bucket: lambda.bucket.name FunctionName: your-function-name Handler: handler.fly
-
Django wsgi lambda function
It wraps up Django and bridge between API gateway and your Django
Requirements:
Put all your dependent packages in
requirements.txt
Optional configuration:
CognitoUserPoolId
: It will set the API gateway authentication if provided. You can only have one perApiBasePath
Configuration:
Lambda: - S3Bucket: lambda.bucket.name Type: wsgi DisableApiGateway: False ApiBasePath: /path FunctionName: your-function-name EnvironmentVariables: DJANGO_SETTINGS_MODULE: mysite.settings
Note:
By default,
DisableApiGateway
is set toFalse
. You must set yourApiBasePath
when it'sFalse
DJANGO_SETTINGS_MODULE
environment variable is a must for it to work -
Cron dispatcher
Cron dispatcher function allows you to create multiple event schedulers on different functions via a single dispatcher. Note: lmdo will construct rule name based on
<user>-<stage>-<service-name>-<FunctionName>--<handler>
. CloudWatch events rule name can only be within 64 characters, so mind your names.Configuration:
Lambda: - S3Bucket: lambda.bucket.name Type: cron_dispatcher FunctionName: your-dispatcher-name RuleHandlers: - Handler: your.module.handler Rate: your cron string e.g. Rate(1 minutes)
They will be replaced with correct value during deployment
$stack|stack-name::output-key
: The value of an existing stack's output based on key name. Can be used both in parameters and templates.
Note:
The stack referring to MUST exist before deployment.
For CommaDelimitedList
type, you can do "$stack|stack-name::key1, $stack|stack-name::key2"
.
To create all functions, run (update/delete similar):
$ lmdo lm create
Options:
--function-name
: Only action on a particular function
To package the function only:
$ lmdo package --function-name=your-function-name
Swagger template is used to create API Gateway
- A folder named 'swagger' under your project folder
- Name your swagger template as
apigateway.json
ApiGatewayName: Your unique Apigateway name
Optionally, you can use ApiVarMapToFile
to map your custom key to a file for replacement
ApiVarMapToFile:
$mappingKey: file/path/name
You can also use ApiVarMapToVar
to map any string values to your defined key. For this mapping, you can also use $stack
and $lambda-arn
utitlity variable for the value.
ApiVarMapToVar:
$mappingKey1: value
$mappingKey2: $stack|stack-name::key1
$mappingKey3: $lambda-arn|lambda-name
NOTE: Please name your version as $version
and your title as $title
so that Lmdo can update it during creation using the value of ApiGatewayName
in your lmdo.yaml
To manage your APIGateway resource, run(update/delete similar):
$ lmdo api create
You can create or delete a stage by running:
$ lmdo api create-stage <from_stage> <to_stage>
or
$ lmdo api delete-stage <from_stage>
CloudWatchEvent:
- Name: rule_name
ScheduleExpression: [schedule-expression]
EventPatternFile: [path/to/pattern/file]
Targets:
- Type: default
Arn: aws-resource-arn
- Type: local
FunctionName: local-function-name
[schedule-expression]
: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/ScheduledEvents.html
[path/to/pattern/file]
: http://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatchEventsandEventPatterns.html
Options:
Description
: description of your rule
DisablePrefix
: default to False
. If True
, lmdo will use your rule name instead of using <user>-<stage>-<service-name>-<rule_name>
RoleArn
: default lmdo will create lambda invokable role
Note
When target type is local
, lmdo will replace it with your function ARN
You can tail any AWS cloudwatch group logs by running:
$ lmdo logs tail your_log_group_name
--day
: defines how many days ago the logs need to be retrieved
--start-date
or --end-date
: specify a start date and/or end date for the log entries using format YYYY-MM-DD
You can also tail logs of your lambda function in your project by running:
$ lmdo logs tail function your-function-name
your-function-name
is the function name you configure in your lmdo.yaml
lmdo offers a simple command line to upload your local static asset into a S3 bucket.
AssetS3Bucket: your.bucket.url
AssetDirectory: directory/where/your/assets/in
To upload (Note: it doesn't delete files):
$ lmdo s3 sync