Running smoke test after environment is healthy

[sheakelly]

Hello,

We had a scenario where beanstalk created a new auto scaling group, the smoke test ran against the old inactive ec2 instance and then eb_deployer swapped the CNAME before the new ec2 instance had started. The result was a period of downtime. Looking at the code the smoke test is run after the environment is updated but before the environment is healthy. Does is make sense to run the smoke test after the environment is healthy?

https://github.com/ThoughtWorksStudios/eb_deployer/blob/master/lib/eb_deployer/eb_environment.rb#L32

[2016-12-13 23:43:44 UTC][package:my-app.zip] start uploading to s3 bucket nib-prod-beanstalk.packages...
[2016-12-13 23:43:48 UTC][package:my-app.zip] uploading finished
[2016-12-13 23:43:48 UTC][application:my-app] Create application version with label master-xxx
[2016-12-13 23:43:49 UTC][cloud_formation_provisioner] No updates are to be performed.
[2016-12-13 23:43:50 UTC][environment:prod-a-8349d9e] Environment update is starting.
[2016-12-13 23:44:00 UTC][environment:prod-a-8349d9e] Updating environment prod-a-8349d9e's configuration settings.
[2016-12-13 23:44:17 UTC][environment:prod-a-8349d9e] Created Auto Scaling launch configuration named: awseb-e-vnbcufykze-stack-AWSEBAutoScalingLaunchConfiguration-1ACH12DFS8ZUQ
[2016-12-13 23:44:53 UTC][environment:prod-a-8349d9e] Deleted Auto Scaling launch configuration named: awseb-e-vnbcufykze-stack-AWSEBAutoScalingLaunchConfiguration-VSYJ244Q9GOK
[2016-12-13 23:44:56 UTC][environment:prod-a-8349d9e] New application version was deployed to running EC2 instances.
[2016-12-13 23:44:56 UTC][environment:prod-a-8349d9e] Successfully deployed new configuration to environment.
[2016-12-13 23:44:56 UTC][environment:prod-a-8349d9e] Environment update completed successfully.
[2016-12-13 23:45:11 UTC][environment:prod-a-8349d9e] running smoke test for my-app-prod-master-prod-inactive.ap-southeast-2.elasticbeanstalk.com...
[2016-12-13 23:45:14 UTC][environment:prod-a-8349d9e] smoke test succeeded.
[2016-12-13 23:45:14 UTC][environment:prod-a-8349d9e] health status: Green
[2016-12-13 23:45:14 UTC][environment:prod-b-d62ce2b] Swap CNAME with env prod-a-8349d9e
[2016-12-13 23:45:12 UTC][environment:prod-b-d62ce2b] swapEnvironmentCNAMEs is starting.
[2016-12-13 23:45:12 UTC][environment:prod-b-d62ce2b] Swapping CNAMEs for environments 'prod-b-d62ce2b' and 'prod-a-8349d9e'.