[ECS] Blue/Green Not Deploying (Primary taskset tg must be behind listener)

0

Hello,

I'm using ECS (Fargate) and blue/green deployment.

Our last successful deployment was on Jan 28, 2020 12:05 PM (UTC-8:00). An attempt to deploy was made last night at Jan 30, 2020 6:56 PM (UTC-8:00) which failed due to the following error:

The ELB could not be updated due to the following error: Primary taskset target group must be behind listener <elb arn>  

This ALB has multiple listeners and services and has had no changes and no issues recently. This deployment configuration hasn't been changed in months and deployments have been routine. Since this began all of my deployment steps just say Skipped and i cannot get new code to production. Luckily, the current ecs task is healthy and fine, but this means i cannot get new code out right now which is not good at all.

I think this is a bug in ALB or CodeDeploy and i think it might be related to this post https://forums.aws.amazon.com/thread.jspa?threadID=316241.

Any help would be great! Thank you

Regards

Edited by: kenndel on Jan 31, 2020 11:10 AM

kenndel
asked 4 years ago4623 views
9 Answers
0

Hi kenndel,

I am sorry you are experiencing this issue. We are aware of the problem and some regions are resolved. Could you please share what region your application is running on?

Thanks,

Kia-AWS
answered 4 years ago
0

We are in us-east-1.

kenndel
answered 4 years ago
0

Thank you for providing your region. As I am looking into the error message further it seems you're green target group is serving traffic before your deployment takes place.

When performing a blue/green deployment, CodeDeploy is expecting to update the traffic on your green target group from 0% to 100%. Thus the production listener for your primary (blue) target group should be the only one receiving traffic before you execute the blue/green deployment. It is failing because it has determined traffic already exists.

Can you confirm if this is your setup and if that was intentional?

Thanks,
Kia

Kia-AWS
answered 4 years ago
0

Thanks for your response! I can confirm that this is my setup but it was not intentional. This configuration was set up several months ago and we haven't really touched it since then, besides adding a few more listeners to the load balancer itself. Typically, codedeploy will be responsible for shifting the traffic between blue and green and i've never manually adjusted it, i'm not even sure how i'd do that without codedeploy! Any solutions to this would be greatly appreciated, thank you!

kenndel
answered 4 years ago
0

Hi kenndel

Thank you for letting me know, the following link can help walk you through Load Balancer Set Up, Target Groups, and Listeners for blue/green ECS Deployments:

https://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-groups-create-load-balancer-for-ecs.html

Let me know if you still need, I am happy to setup a call to walk you through the changes.

Thanks
Kia

Kia-AWS
answered 4 years ago
0

Hi Kia, I've reviewed those docs again and i don't think they help my situation. I also have a support ticket in for this issue but I'd love to hop on a call to discuss the fix!

kenndel
answered 4 years ago
0

Please check your Private Messages so I can connect with you directly and help you resolve your configuration.

Kia-AWS
answered 4 years ago
0

This issue has been resolved via a support ticket. I'll be posting the resolution here if anyone comes across it. The problem was, at some point during a deployment, traffic was moved 100% to my Green TG, my Blue TG was set to 0% traffic, but Blue TG was not removed from the load balancer rule. As a result, when i began a new deployment it noticed i had rules for both TGs and backed down. Pretty simple fix but, as i've never looked at the LB for this reason since getting Blue/Green deployments set up i never noticed that the blue TG at 0% was actually a problem, i assumed it just scaled them back and forth from 0 to 100 as needed. I hope this helps someone in the future, here are the steps:

Next Steps

As a workaround, please consider to make modifications on load balancer listener rules by following the below steps.

1. Modify the load balancer listener rules "lb rule arn" and "other lb rule arn" to point to **target group "green tg name" only and also delete the weight value for target group.**  

2. Modify the load balancer listener default rule "lb default rule arn" to point to target group "green tg name".  
  
3. Create a new deployment based on the same deployment group.  

Edited by: kenndel on Feb 11, 2020 12:43 PM

kenndel
answered 4 years ago
0

The ELB could not be updated due to the following error: Green taskset target group cannot have non-zero weight prior to traffic shifting on listener.
80 and 443 ports are listeners and the target groups both are forwarding to port number 80. listener 80 is forwarding to tg-2 and 443 is forwarding to tg-1 in ecs(blue/green) deployment.while updating with new revision number in ecs the revision number is not updating .the deploy stays in blue task set only not moving to green one

prathap
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions