Skip to content

Bug report on ECS Built-In Blue/Green functionality

3

After seeing this blog post I was very excited to give this new deployment option a try. Turns out that once you update an ECS service to use strategy BLUE_GREEN, subsequent updates to go back to a ROLLING strategy fail with a very generic error deployment failed: hook execution failure(s) detected. In my ecs update-service call, I tried setting lifecycleHooks to an empty array [], as well as giving it a valid lambda function for the "PRE_SCALE_UP" hook. Both failed and rolled back to the previous stable blue/green deployment.

One other commenter on the blog post seemed to note this issue(Kemal), but didn't provide any details, so it's hard to know exactly what they were seeing on their specific deployment events.

For additional context, my blue/green deployment uses a load balancer with two target groups. The rolling deployment config uses the same lb with one target group. The only way I can use ROLLING strategy deployments right now is if I re-create my service and only use ROLLING strategy. The moment I switch to BLUE_GREEN, it does not let me get any successful ROLLING deployments out.

Is anyone else seeing this behavior on their accounts?

asked 10 months ago471 views
2 Answers
0

The issue you're experiencing with ECS built-in blue/green deployments appears to be related to the transition between deployment strategies. When switching from a BLUE_GREEN deployment strategy back to a ROLLING strategy, the error "deployment failed: hook execution failure(s) detected" suggests there might be issues with the lifecycle hooks configuration.

This behavior is likely occurring because the blue/green deployment strategy uses specific lifecycle hooks and deployment controller configurations that need to be properly addressed when switching back to a rolling deployment. The blue/green deployment strategy in Amazon ECS is designed to run two identical production environments and validate new service revisions before directing production traffic to them, which requires different configuration than a rolling deployment.

When you attempt to switch back to a ROLLING strategy, the service might still be referencing lifecycle hooks or other blue/green specific configurations that aren't compatible with the rolling deployment model, causing the deployment to fail.

To resolve this issue, you might need to:

  1. Ensure all lifecycle hooks are properly removed or reconfigured when switching back to a rolling deployment
  2. Consider creating a new service with the desired deployment strategy rather than updating an existing one
  3. Check if there are any lingering resources from the blue/green deployment that might be causing conflicts (such as target groups or load balancer configurations)

Since this appears to be a relatively new feature (based on your reference to a recent blog post), it's possible that there are still some limitations or bugs in the transition between deployment strategies that AWS may need to address.
Sources
Amazon ECS enables built-in blue/green deployments - AWS
Amazon ECS blue/green deployments - Amazon Elastic Container Service
Amazon ECS service deployment controllers and strategies - Amazon Elastic Container Service

answered 10 months ago
  • According to this document it should be possible to migrate from ECS Blue/green to rolling.

0

For anyone that stumbles upon the same problem, I can confirm AWS has resolved this issue.

answered 9 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.