Feature/dynamo state

.bumpversion.cfg

@@ -14,6 +14,6 @@ replace = __version__ = {new_version}
 
 [semver]
 main_branches = development
-major_branches =
+major_branches = release, major
 minor_branches = feature
 patch_branches = hotfix, bugfix

README.md

@@ -13,20 +13,25 @@ pip install path/to/deployer-<version>.tar.gz
 Deployer is free for use by RightBrain Networks Clients however comes as is with out any guarantees.
 
 ##### Flags
-* -c --config <config file> (REQUIRED) : Yaml configuration file to run against.
+* -c --config <config file> : Yaml configuration file to run against.
 * -s --stack <stack name>  (REQUIRED) : Stack Name corresponding to a block in the config file.
 * -x --execute <execute command> (REQUIRED) : create|update|delete|sync|change Action you wish to take on the stack.
 * -p --profile <profile>     : AWS CLI Profile to use for AWS commands [CLI Getting Started](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html).
 * -P --param <PARAM>, --param PARAM An override for a parameter
+* -J --json-param <JSON_PARAM_STRING> :A JSON string for overriding a collection of parameters
 * -y --copy <copy>        : Copy directory structures specified in the configuration file under the sync_dirs configuration.
 * -A --all         : Create or Update all stacks in the configuration file, Note you do not have to specify a stack when using this option.
-* -r --disable-roleback : Disable rollback on failure, useful when trying to debug a failing stack.
+* -r --disable-rollback : Disable rollback on failure, useful when trying to debug a failing stack.
 * -t --timeout : Sets Stack create timeout
 * -e --events : Display events of the CloudFormation stack at regular intervals.
-* -z --zip         : Pip install requirements, and zip up lambda's for builds.
+* -z --zip-lambas         : Pip install requirements, and zip up lambda's for builds.
 * -t --change-set-name <change set name> (REQUIRED Change Sets Only) : Used when creating a change set to name the change set.
 * -d --change-set-description <change set description> (REQUIRED Change Sets Only) : Used when creating a change set to describe the change set.
-* -j, --assume-valid    Assumes templates are valid and does not do upstream validation (good for preventing rate limiting)
+* -j --assume-valid   : Assumes templates are valid and does not do upstream validation (good for preventing rate limiting)
+* -O --export-yaml <EXPORT_YAML_FILE> : Export stack config to specified YAML file.
+* -o --export-json <EXPORT_JSON_FILE> : Export stack config to specified JSON file.
+* -i --config-version <CONFIG_VERSION_ACTION> : Execute ( list | get | set ) of stack config.
+* -n --config-version-number <CONFIG_VERSION_NUMBER> : Specified config version, used with --config-version option.
 * -D, --debug           Sets logging level to DEBUG & enables traceback
 * -v, --version         Print version number
 * --init [INIT]         Initialize a skeleton directory
@@ -58,6 +63,7 @@ Zip up lambdas, copy to s3, and update.
 *Note* See [example_configs/dev-us-east-1.yml](./example_configs/dev-us-east-1.yml) for an example configuration file.
 
 The config is a large dictionary. First keys within the dictionary are Stack Names. The global Environment Parameters is a common ground to deduplicate parameter entries that are used in each Stack. Stack parameters overwrite global parameters. 
+When deployer is run, it creates a DynamoDB Table called CloudFormation-Deployer if it does not already exist. The stack configuration in the config file is saved into DynamoDB, and any future changes result in a new entry with an updated timestamp. 
 
 ## Required
 The following are required for each stack, they can be specified specifically to the stack or in the global config.
@@ -122,6 +128,32 @@ These parameters provide identity to the Services like what AMI to use or what b
       UploadInstanceType: t2.medium
 ```
 
+Parameters can be overridden from the command line in several different ways. 
+
+The first (which takes precedence) is the -P option. Parameters can be specified in the following form:
+```
+deployer -P 'Param1=Value1'
+``` 
+deployer will set the value of parameter 'Param1' to 'Value1', even if it is also specified in the config file. -P can be specified multiple times for multiple parameters
+
+The second is the -J option. Parameters can be specified in the following form:
+```
+deployer -J '{"Param1":"Value1"}'
+```
+This option allows the user to specify multiple parameter values as a single JSON object. This will override parameters of the same name as those specified in the config file as well.
+
+It is important to note that since a stack's configuration is saved in the DynamoDB state table, specifying these overrides without sending a config file will use the existing configuration for the stack retrieved from the table, but with the overridden parameter values swapped in.
+If it is desirable to send a config file to update some of the parameter values but keep some of the existing values from the previous configuration, it can be done like this:
+```
+    parameters:
+      Monitoring: 'True'
+      NginxAMI:
+        UsePreviousValue: True
+      NginxInstanceType: t2.medium
+```
+Notice that for the NginxAMI parameter, the value is now a dictionary instead of a string, and the UsePreviousValue key is set to True. This indicates to deployer to use the existing value in the configuration for the NginxAMI parameter.
+
+
 ## Lookup Parameters
 
 These are parameters that can be pulled from another stack's output. `deployer` tolerates but logs parameters that exist within the configuration but do not exist within the template.
@@ -172,6 +204,30 @@ Denote that at tranform is used in a stack and deployer will automatically creat
   transforms: true
 ```
 
+## Versions
+There are several command line options that allow the user to view and set the configuration based on version number.
+
+When a new configuration is saved automatically to the DynamoDB table, a version number is generated and assigned to it. These versions can be viewed like this:
+```
+./deployer -s <Stack Name> --config-version list
+```
+This will output a list of config version numbers and creation timestamps. Viewing a specific configuration based on the number can be done like this:
+```
+./deployer -s MyStack --config-version get --config-version-number 1
+```
+In the above example, the output will return the configuration for stack MyStack with the version number 1, the original configuration for the stack. We can then effectively roll back to that configuration with this command:
+```
+./deployer -s MyStack --config-version set --config-version-number 1
+```
+This will set the configuration for MyStack back to version 1, reverting the values for parameters, tags, etc.  
+
+## Exports
+The configuration for a stack can be exported to a file as well. Two formats are supported, JSON and YAML. An example for each is shown here:
+```
+./deployer -s MyStack --export-yaml ../mystack-config.yaml
+./deployer -s MyStack --export-json configs/mystack-config.json
+```
+
 ## Updates
 When running updates to a stack you'll be running updates to the CloudFormation Stack specified by Stack. 
 
@@ -225,7 +281,7 @@ Currenly there is only the Stack class, Network and Environment classes are now
 This is the class that builds zip archives for lambdas and copies directories to s3 given the configuration in the config file.
 
 **Note** 
-Network Class has been removed, it's irrelivant now. It was in place because of a work around in cloudformation limitations. The abstract class may not be relivant, all of the methods are simmular enough but starting this way provides flexablility if the need arise to model the class in a different way. 
+Network Class has been removed, it's irrelevant now. It was in place because of a work around in cloudformation limitations. The abstract class may not be relivant, all of the methods are simmular enough but starting this way provides flexablility if the need arise to model the class in a different way. 
 
 
 # Config Updater
@@ -353,3 +409,29 @@ Our top template contains numerous references to child templates. Using a combin
 ```
 
 You can add your own templates under the `cloudformation` directory to deploy your own stacks. Each stack will also need an entry in your deployer config file to specify which directories should be uploaded, the name of the stack, and any required parameters.
+
+# Upgrade path to 1.0.0
+
+A breaking change is made in the 1.0.0 release. The stack_name attribute in the stack configuration is now deprecated. The resulting CloudFormation stack that is created is now the name of the stack definition. For example, consider the following stack definition:
+
+```
+deployer:
+  stack_name: shared-deployer
+  template: cloudformation/deployer/top.yaml
+  parameters: 
+    Environment: Something
+```
+
+In previous versions, the CloudFormation stack that gets deployed from this is called `shared-deployer`. In 1.0.0+, the CloudFormation stack that gets deployed is called `deployer`. 
+
+This means that for existing configurations, the top level stack definition name must be changed to match the stack_name attribute, like this:
+
+```
+shared-deployer:
+  stack_name: shared-deployer
+  template: cloudformation/deployer/top.yaml
+  parameters: 
+    Environment: Something
+``` 
+
+This will ensure that deployer recognizes the existing CloudFormation stack, rather than forcing you to create a new one.

deployer/__init__.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python
 import argparse
 import json
+import yaml
 import os
 from botocore.exceptions import ClientError
 from deployer.stack import Stack
@@ -24,10 +25,11 @@
 def main():
     # Build arguement parser
     parser = argparse.ArgumentParser(description='Deploy CloudFormation Templates')
-    parser.add_argument("-c", "--config", help="Path to config file.")
+    parser.add_argument("-c", "--config", help="Path to config file.",default=None)
     parser.add_argument("-s", "--stack", help="Stack Name.")
     parser.add_argument("-x", "--execute", help="Execute ( create | update | delete | upsert | sync | change ) of stack.")
     parser.add_argument("-P", "--param", action='append', help='An override for a parameter')
+    parser.add_argument("-J", "--json-param", help='A JSON string for overriding a collection of parameters')
     parser.add_argument("-p", "--profile", help="Profile.",default=None)
     parser.add_argument("-t", "--change-set-name", help="Change Set Name.")
     parser.add_argument("-d", "--change-set-description", help="Change Set Description.")
@@ -40,6 +42,10 @@ def main():
     parser.add_argument("-D", "--debug", help="Sets logging level to DEBUG & enables traceback", action="store_true", dest="debug", default=False)
     parser.add_argument("-v", "--version", help='Print version number', action='store_true', dest='version')
     parser.add_argument("-T", "--timeout", type=int, help='Stack create timeout')
+    parser.add_argument("-O", "--export-yaml", help="Export stack config to specified YAML file.",default=None)
+    parser.add_argument("-o", "--export-json", help="Export stack config to specified JSON file.",default=None)
+    parser.add_argument("-i", "--config-version", help="Execute ( list | get | set ) of stack config.")
+    parser.add_argument("-n", "--config-version-number", help="Specified config version, used with --config-version option.")
     parser.add_argument('--init', default=None, const='.', nargs='?', help='Initialize a skeleton directory')
     parser.add_argument("--disable-color", help='Disables color output', action='store_true', dest='no_color')
 
@@ -71,10 +77,19 @@ def main():
     # Validate arguements and parameters
     options_broken = False
     params = {}
-    if not args.config:
-        args.config = 'config.yml'
+    if args.all:
+        if not args.config:
+            print(colors['warning'] + "Must Specify config flag!" + colors['reset'])
+            options_broken = True
     if not args.all:
-        if not args.execute:
+        if args.config_version:
+            if args.config_version != "list" and args.config_version != "set" and args.config_version != "get":
+                print(colors['warning'] + "config-version command '" + args.config_version + "' not recognized. Must be one of: list, set, get "+ colors['reset'])
+                options_broken = True
+            if (args.config_version == 'set' or args.config_version == 'get') and not args.config_version_number:
+                print(colors['warning'] + "config-version " + args.config_version + " requires config-version-number flag!" + colors['reset'])
+                options_broken = True
+        elif not args.execute:
             print(colors['warning'] + "Must Specify execute flag!" + colors['reset'])
             options_broken = True
         if not args.stack:
@@ -89,6 +104,20 @@ def main():
                 print(colors['warning'] + "Invalid format for parameter '{}'".format(param) + colors['reset'])
                 options_broken = True
 
+    try:
+        json_param_dict = {}
+        if args.json_param:
+            json_param_dict = json.loads(args.json_param)
+            if args.param:
+                #Merge the dicts
+                merged_params = {**json_param_dict, **params}
+                params = merged_params
+            else:
+                params = json_param_dict
+    except:
+        print(colors['warning'] + "Invalid format for json-param, must be valid json." + colors['reset'])
+        options_broken = True
+
     # Print help output
     if options_broken:
         parser.print_help()
@@ -100,33 +129,90 @@ def main():
         console_logger.setLevel(logging.ERROR)
 
     try:
-        # Read Environment Config
-        with open(args.config) as f:
-            config = ruamel.yaml.safe_load(f)
 
         # Load stacks into queue
         stackQueue = []
         if not args.all:
             stackQueue = [args.stack]
         else:
-            for stack in config.items():
+            #Load config, get stacks
+            try:
+                with open(args.config) as f:
+                    file_data = ruamel.yaml.safe_load(f)
+            except Exception as e:
+                msg = str(e)
+                logger.error("Failed to retrieve data from config file {}: {}".format(file_name,msg))
+                exit(3)
+
+            for stack in file_data.keys():
                 if stack[0] != "global":
-                    stackQueue = find_deploy_path(config, stack[0], stackQueue)
+                    stackQueue = find_deploy_path(config_object.get_config(), stack[0], stackQueue)
 
         # Create or update all Environments
         for stack in stackQueue:
             if stack != 'global' and (args.all or stack == args.stack):
 
                 logger.info("Running " + colors['underline'] + str(args.execute) + colors['reset'] + " on stack: " + colors['stack'] + stack + colors['reset'])
-
+
+                # Create deployer config object
+                cargs = {
+                  'profile': args.profile,
+                  'stack_name': stack
+                }
+                if args.config:
+                    cargs['file_name'] = args.config
+
+                if args.param or args.json_param:
+                    cargs['override_params'] = params
+
+                config_object = Config(**cargs)
+
+                #Config Version Handling
+                if args.config_version:
+                    if args.config_version == "list":
+                        versions = config_object.list_versions()
+                        for version in versions:
+                            if 'version' in version:
+                                print("Timestamp: {}  Version: {}".format(version['timestamp'], version['version']))
+                    elif args.config_version == "get":
+                        retrieved_config = config_object.get_version(args.config_version_number)
+                        print(yaml.dump(retrieved_config,default_flow_style=False, allow_unicode=True))
+                    elif args.config_version == "set":
+                        config_object.set_version(args.config_version_number)
+
+                    continue
+
+                #Export if specified
+                if args.export_json:
+                    config_dict = config_object.get_config()
+
+                    try:
+                        with open(args.export_json, 'w') as f:
+                            j = json.dumps(config_dict, indent=4)
+                            f.write(j)
+                    except Exception as e:
+                        msg = str(e)
+                        logger.error("Failed to export data to JSON file {}: {}".format(args.export_json,msg))
+                        exit(3)
+
+                if args.export_yaml:
+                    config_dict = config_object.get_config()
+
+                    try:
+                        with open(args.export_yaml, 'w') as f:
+                             yaml.dump(config_dict, f, default_flow_style=False, allow_unicode=True)
+                    except Exception as e:
+                        msg = str(e)
+                        logger.error("Failed to export data to YAML file {}: {}".format(args.export_yaml,msg))
+                        exit(3)
+
                 # Build lambdas on `-z`
                 if args.zip_lambdas:
                     logger.info("Building lambdas for stack: " + stack)
-                    LambdaPrep(args.config, args.stack).zip_lambdas()
-
-                # Create deployer config object
-                config_object = Config(args.config, stack)
-
+                    lambda_dirs = config_object.get_config_att('lambda_dirs', [])
+                    sync_base = config_object.get_config_att('sync_base', '.')
+                    LambdaPrep(sync_base, lambda_dirs).zip_lambdas()
+
                 # AWS Session object
                 session = Session(profile_name=args.profile, region_name=config_object.get_config_att('region'))
 
@@ -141,15 +227,14 @@ def main():
 
                 # S3 bucket to sync to
                 bucket = CloudtoolsBucket(session, config_object.get_config_att('sync_dest_bucket', None))
-
+                
                 # Check whether stack is a stack set or not and assign corresponding object
                 if(len(config_object.get_config_att('regions', [])) > 0 or len(config_object.get_config_att('accounts', [])) > 0):
                     env_stack = StackSet(session, stack, config_object, bucket, arguements)
                 else:
                     if args.timeout and args.execute not in ['create', 'upsert']:
                         logger.warning("Timeout specified but action is not 'create'. Timeout will be ignored.")
                     env_stack = Stack(session, stack, config_object, bucket, arguements)
-
                 try:
 
                     # Sync files to S3
@@ -185,6 +270,7 @@ def main():
         if args.debug:
             tb = sys.exc_info()[2]
             traceback.print_tb(tb)
+        exit(1)
 
 def find_deploy_path(stackConfig, checkStack, resolved = []):
     #Generate depedency graph

deployer/cloudformation.py

@@ -1,5 +1,4 @@
 #!/usr/bin/env python
-import git
 from abc import ABCMeta, abstractmethod
 from deployer.logger import logger
 
@@ -52,20 +51,6 @@ def reload_stack_status(self):
     def status(self):
         pass
 
-    def get_repository(self, base):
-        try:
-            return git.Repo(base, search_parent_directories=True)
-        except git.exc.InvalidGitRepositoryError:
-            return None
-
-    def get_repository_origin(self, repository):
-        try:
-            origin = repository.remotes.origin.url
-            return origin.split('@', 1)[-1] if origin else None
-        except (StopIteration, ValueError):
-            return None
-        return None
-
     def get_template_body(self, bucket, template):
         if not bucket:
             try: