blue-green hangs on 'errors' in update env operation?

[mepatterson]

So I'm using eb_deployer to push an app with a Dockerfile up to EB. EB then handles all the spin-up of Docker containers and all that. Works great. But when I started using blue-green for my production deploys, I ran into this issue: once I have a green, running environment and a red, inactive environment (red because the instance has been killed so health check is failing, but that's okay), deploys with eb_deployer will hang at:

[2016-02-23 03:34:56 UTC][environment:prod-b-9cf234d] Successfully deployed new configuration to environment.
[2016-02-23 03:34:57 UTC][environment:prod-b-9cf234d] Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.

The 'errors', as far as I can tell, are actually because it's just taking a while to spin up a whole new instance and rebuild the docker container from scratch. But that's okay with me. If I wait for a bit longer, on the EB side, the deploy finishes fine and it switches from red to green. But eb_deployer never seems to catch on that this has (finally) happened, and it just hangs there forever, so it never does the rest of the magic to tell EB to switch cnames and set inactive and all that.

Any ideas on how I could resolve this? Is there a setting in eb_deployer, or a timeout or something I can change?

[wpc]

hi, @mepatterson
Can you add a --debug option and past the full deploy log here. The hanging behavior is not acceptable, I need figure out where exactly it happens.

-- wpc

[mepatterson]

Sorry, I ended up switching to phoenix mode for my docker-based deploys and
I haven't been able to reproduce this issue since. It might have been
something unique to my configuration of eb_deployer. I'll add more details
if I see this happen on any future deploys I setup.

On Thu, Feb 25, 2016 at 11:52 AM WPC [email protected] wrote:

hi, @mepatterson https://github.com/mepatterson
Can you add a --debug option and past the full deploy log here. The
hanging behavior is not acceptable, I need figure out where exactly it
happens.

-- wpc

—
Reply to this email directly or view it on GitHub
#69 (comment)
.

[harryherbig]

Same problem here. Deployment with eb_deployer fails, if inactive has 0 instances:

2016-04-27 15:31:58 UTC+0200 INFO Environment health has transitioned from RED to GREEN

2016-04-27 15:28:56 UTC+0200 ERROR Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.

2016-04-27 15:28:56 UTC+0200 INFO Successfully deployed new configuration to environment.

2016-04-27 15:28:56 UTC+0200 INFO New application version was deployed to running EC2 instances.

2016-04-27 15:28:22 UTC+0200 INFO Command execution completed on all instances successfully.

2016-04-27 15:28:21 UTC+0200 INFO Deploying new version to instance(s).

2016-04-27 15:26:53 UTC+0200 INFO Adding instance 'i-XXXXXX' to your environment.

2016-04-27 15:26:52 UTC+0200 INFO Added EC2 instance 'i-XXXXX' to Auto Scaling Group 'XXXXX'.

2016-04-27 15:26:28 UTC+0200 INFO Updating environment production-a-XXXX configuration settings.

2016-04-27 15:26:20 UTC+0200 INFO Environment update is starting.

[wpc]

@harryurban thanks for reporting this. I will take a look

[harryherbig]

Even increasing the Command Timeout at elasticbeanstalk doesnt help:
- namespace: aws:elasticbeanstalk:command option_name: Timeout value: "1000"

tried it with our CI Server and from my machine with --debug option. -> no difference
I guess spinning up the first instance takes to long?

[Jwata]

Same issue here. Any update for this?
If not, I'll try to have some waiting time for it.

I'm thanksfully using eb deployer for our projects, but the issue happens quit often.
When the number of instance is zero, I think it always happens.
Then I retry deployment again and it successes.

[wpc]

I found the issue. Will release a fix in 0.6.7 soon

[dgolombek]

Any further updates on this? I see that #79 was merged, but that simply handles the error and aborts -- did you find a root cause here?

[ghost]

I'm seeing this issue all the time. Any update on the fix?

[fkelly]

We're also very keen to get this fixed. I'm sure you're busy ... any chance you want to describe how you'd do the fix and one of us can implement it and do a pull request? Thanks!

[ghost]

So looks like elasticbeanstalk doen't support the configuration of asg MinMax 0. So when we do a deployment on a blue-green stack and blue has 0 instances and you deploy code. It'll update the asg to MinMax that you set but doesn't wait till the ec2 instances is up and ready before it says that its updated the code for ec2 instances. So you get that error. So we'll never get that success message the eb_deployer is looking for. The fix seems to be listening for another common message.

If you listen for

"Successfully deployed new configuration to environment"

this seems to fix the problem. The smoke test is initiated and we just keep polling till the ec2 instance is up and running

[ghost]

here's the pull request #86

[dcrockwell]

Can confirm this still seems to be an issue. Waiting for #86 to be merged in.

New application version was deployed to running EC2 instances.
Successfully deployed new configuration to environment.
Update environment operation is complete, but with errors. For more information, see troubleshooting documentation.

Then, after a few more minutes:

Environment health has transitioned from RED to GREEN