ec2 image cloud-init docker startup issue

[dmytroleonenko]

Output of docker version:

Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.centos.14.x86_64
 Go version:      go1.6.3
 Git commit:      cb079f6-unsupported
 Built:           Fri Sep 16 13:24:25 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.centos.14.x86_64
 Go version:      go1.6.3
 Git commit:      cb079f6-unsupported
 Built:           Fri Sep 16 13:24:25 2016
 OS/Arch:         linux/amd64

Output of docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: atomicos-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 10.74 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 826.8 MB
 Data Space Total: 20.07 GB
 Data Space Available: 19.24 GB
 Metadata Space Used: 192.5 kB
 Metadata Space Total: 54.53 MB
 Metadata Space Available: 54.33 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2016-06-09)
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: null host bridge
Kernel Version: 3.10.0-327.36.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 1
Total Memory: 991.6 MiB
Name: ip-10-216-10-43.ec2.internal
ID: M6XS:4YA3:PQCG:3U4A:5REJ:LF3C:FRU2:5ARN:CC63:3ZC7:BMJ7:HOWJ
Registries: docker.io (secure)

Additional environment details (AWS, VirtualBox, physical, etc.):

• AWS VPC
• us-east-1 region
• official ami-cfeca0d8 AMI

Steps to reproduce the issue:

1. start an instance with the following/similar user-data:

#cloud-configs

write_files:
- path: /etc/systemd/system/docker-zk-exhibitor.service
  permissions: 0644
  owner: root
  content: |
    [Unit]
    Description=Run a zk-exhibitor container
    Requires=docker.service
    After=docker.service

    [Service]
    Restart=always
    TimeoutStartSec=30m
    ExecStartPre=/usr/bin/bash -c 'eval $(/usr/bin/docker run --rm dockerhubtungsten/awscli ecr get-login --region us-east-1)'
    ExecStartPre=/usr/bin/docker pull myaccount.dkr.ecr.us-east-1.amazonaws.com/tn/zk-exhibitor:latest
    ExecStartPre=-/usr/bin/docker rm zk-exhibitor
    ExecStart=/usr/bin/docker run --rm -p 8181:8181 -p 2181:2181 -p 2888:2888 -p 3888:3888 --name zk-exhibitor -e S3_BUCKET=mybucket-us-east-1 -e S3_PREFIX=1_ -e AWS_REGION=us-east-1 -e HOSTNAME=%H myaccount.dkr.ecr.us-east-1.amazonaws.com/tn/zk-exhibitor:latest
    ExecStop=/usr/bin/docker stop -t 2 zk-exhibitor

    [Install]
    WantedBy=multi-user.target

runcmd:
- systemctl daemon-reload
- systemctl enable docker-zk-exhibitor.service
- systemctl start docker.service
- systemctl start docker-zk-exhibitor.service

Describe the results you received:
docker daemon is not running when the instance starts. It's not possible to start it with systemctl start docker.service because it hangs
Nothing in journalctl -u docker.service
The way to make it to start is actually to do the following steps

1. lvremove atomicos/docker-pool -y
2. rm -rf /var/lib/docker/*
3. docker-storage-setup --reset
4. docker-storage-setup
5. reboot

Describe the results you expected:
docker daemon running upon startup

Additional information you deem important (e.g. issue happens only occasionally):
The issue is somehow related to device-mapper setup. Should be pretty easy to reproduce

[dustymabe]

hey @dmytroleonenko - are you still having trouble? if you just boot the instance without your user-data is docker up and running fine?

@jasonbrooks the image he is using is a centos atomic host:

aws ec2 describe-images --image-ids ami-cfeca0d8 
IMAGES  x86_64  2016-10-06T10:35:10.000Z        CentOS Atomic Host x86_64 HVM EBS 1609  xen     ami-cfeca0d8    410186602215/CentOS Atomic Host x86_64 HVM EBS 1609    machine CentOS Atomic Host x86_64 HVM EBS 1609  410186602215    True    /dev/sda1       ebs     available       hvm
BLOCKDEVICEMAPPINGS     /dev/sda1
EBS     False   False   snap-e9d9285b   10      standard