Refactor datagov-deploy

[SkipKeats]

Per discussion with Hyon and team, datagov-deploy will be refactored into, essentially, the present/past (Vagrant/VIrtualBox) and the future (Docker). The how is under discussion.

Parent to #301,
Parent to #302,
Parent to #303,
Parent to #304,
Parent to #305,
Parent to #306,
Parent to #307,
Parent to #308,
Parent to #309,
Parent to #310,
Parent to #311,
Parent to #312,
Parent to #313,
Parent to #314,
Parent to #315

[SkipKeats]

John, Hareesh, and Skip will work on diagramming refactoring ideas for Datagov-Deploy and compare.

[JJediny]

Here are my thoughts on refactoring this repo:

1. Make a repo for each deployable unit (e.g. catalog, harvester, inventory, dashboard, crm, wordpress, solr, jekyll) and each agent (New Relic APM, New Relic Infrastructure, Nessus, Fluentd, elasticalert)

In each repo structure accordingly using ansible role best practices so they can be pulled in using ansible-galaxy pull to vendor in these broken out roles ^ (Also see ansibleplaybookbundle/ansible-playbook-bundle#269 as a possible way to reuse our ansible work for future kubernetes deployment testing)

# Example for catalog ckan
defaults/
files/
handlers/
meta/main.yml # ansible-galaxy init file to pull in vendored roles needed for application deployment
tasks/
templates/
tests/test.yml  # test playbook to run with circleci
vars/
vendor/  #role dependencies declared in meta/main.yml
.circle/config.yml # circleci v2.0 spec
.kitchen.yml  # test-kitchen recipe to run ansible role in docker container using ansible-kitchen and kitchen-docker (used by circleci)
.kitchen.vm.yml  # test-kitchen recipe to run ansible role in virtualbox using ansible-kitchen and kitchen-vagrant
Gemfile  # for test-kitchen gems to simplfy ruby deployment 
Dockerfile  #  
docker-compose.yml  #docker-compose used for local testing of networked services (redis/mysql/postgres/etc) should use `build: .` to point to the Dockerfile and latest/standard services and port networking.
packer.json  # packer template to run ansible-local provisioner for use on AWS auto-scaling-group/terraform work
.gitignore # .kitchen .vagrant Gemfile.lock

[SkipKeats]

@JJediny: The above is what I was generally thinking; however, how would you deal with the current/past and future situations, i.e., vagrant/virtualbox and docker respectively? The files necessary for each scenario due not entirely match. And with vagrant, each Vagrantfile must be in its own directory, although the Ansible within it could call into another part of the 'tree'. If both Vagrant and Docker were to call to the same Ansible directory, we would need to differentiate yaml files that were specific to each type, e.g., v-datagov-web.yml or d-datagov-web.yml. At present, all of the 'application' yaml files sit at the top Ansible level. Thoughts?

[JJediny]

@SkipKeats the current/past will need to be addressed with the use of variables and when statements in the roles. Also I'm not sure we want to use raw vagrantfiles as they are alot harder to customize, instead I think we should be using the test-kitchen framework to declare application specfic context. test-kitchen has bother a docker driver and vagrant driver and it makes it much easier to develop test cases in the future using the verifier in test-kitchen to right acceptance test using inspec or the like. Thoughts

[SkipKeats]

I will look into the test-kitchen framework.

[JJediny]

https://github.com/neillturner/kitchen-ansible

Example for Docker driver

---
driver:
  name: docker
  use_sudo: false
  privileged: true
  #http_proxy: <%= ENV['http_proxy'] || nil %>
  #https_proxy: <%= ENV['https_proxy'] || nil %>

transport:
  name: sftp
  max_ssh_sessions: 5

provisioner:
  name: ansible_playbook
  roles_path: ../
  #requirements_path: requirements.yml
  hosts: all
  playbook: test.yml
  ansible_verbose: true
  ansible_verbosity: 4
  ansible_extra_flags: <%= ENV['ANSIBLE_EXTRA_FLAGS'] %>
  require_ansible_omnibus: false
  ansible_connection: ssh
  #http_proxy: <%= ENV['http_proxy'] || nil %>
  #https_proxy: <%= ENV['https_proxy'] || nil %>
  #Testing - Busser requires Chef to run test suites
  require_chef_for_busser: false
  require_ruby_for_busser: false

platforms:
  - name: ubuntu-14.04
    driver: 
    image: rndmh3ro/docker-ubuntu1404-ansible:latest   
  - name: ubuntu-16.04
    driver: 
    image: rndmh3ro/docker-ubuntu1604-ansible:latest  
  - name: centos-7
    driver: 
    image: rndmh3ro/docker-centos7-ansible:latest      

suites:
  - name: test

#verifier:
#  name: inspec
#  sudo: true
#  inspec_tests:
#    - https://github.com/dev-sec/linux-baseline

[SkipKeats]

Thank you.

[SkipKeats]

I have been looking at and experimenting with Chef Test-kitchen a bit, as well as looking at the assorted links above related to Ansible, Ansible Galaxy, etc. In my mind, I see a 3D cube of sorts per the refactoring, which consists of the following:

• Virtualization Environment
  • Docker/Ansible/Kitchen (Future)
  • Vagrant-VirtualBox/Ansible/Kitchen (Current)
• Server Environment
  • Development
  • Testing
  • Staging
  • Production
• Projects/Applications -- In separate repos, then leveraged into each virtualization and server schema.
  • Python/CKAN:
    • Catalog-app, and
    • Inventories
  • PHP
    • CodeIgniter
      • Dashboard
      • CRM
    • WordPress

If I am missing an application, please let me know.

The current roles, as currently construed, would be split across these divisions. I am still trying to figure out how to deal with both Vagrant and Docker in the same primary repository, because I have not found any documentation that says it is allowable. I would think that we would need a primary file, probably a kitchen.yml, that would differentiate between current and future via switches of some form when run. I just not certain how to do that at present.

Thoughts would be appreciated.

[SkipKeats]

John and Hareesh: FIrst draft of repositories per refactoring. Just want to make certain we are on the same page. Services and like not included yet and, per John's suggestion, will not go into the refactoring process in first go, e.g., kibana, etc.
refactor_repositories_datagov-deploy.pdf

[SkipKeats]

Per John's request, a SVG version of the refactor diagram is contained in the attached zip file. The primary file is DDRP.svg, but all of the icons are separate files.
refactor_repositories_datagov-deploy.zip

[SkipKeats]

All initial repositories are now created.