[ECS] Blue/Green Not Deploying (Primary taskset tg must be behind listener)

0

Hello,

I'm using ECS (Fargate) and blue/green deployment.

Our last successful deployment was on Jan 28, 2020 12:05 PM (UTC-8:00). An attempt to deploy was made last night at Jan 30, 2020 6:56 PM (UTC-8:00) which failed due to the following error:

The ELB could not be updated due to the following error: Primary taskset target group must be behind listener <elb arn>  

This ALB has multiple listeners and services and has had no changes and no issues recently. This deployment configuration hasn't been changed in months and deployments have been routine. Since this began all of my deployment steps just say Skipped and i cannot get new code to production. Luckily, the current ecs task is healthy and fine, but this means i cannot get new code out right now which is not good at all.

I think this is a bug in ALB or CodeDeploy and i think it might be related to this post https://forums.aws.amazon.com/thread.jspa?threadID=316241.

Any help would be great! Thank you

Regards

Edited by: kenndel on Jan 31, 2020 11:10 AM

kenndel
질문됨 4년 전4725회 조회
9개 답변
0

Hi kenndel,

I am sorry you are experiencing this issue. We are aware of the problem and some regions are resolved. Could you please share what region your application is running on?

Thanks,

Kia-AWS
답변함 4년 전
0

We are in us-east-1.

kenndel
답변함 4년 전
0

Thank you for providing your region. As I am looking into the error message further it seems you're green target group is serving traffic before your deployment takes place.

When performing a blue/green deployment, CodeDeploy is expecting to update the traffic on your green target group from 0% to 100%. Thus the production listener for your primary (blue) target group should be the only one receiving traffic before you execute the blue/green deployment. It is failing because it has determined traffic already exists.

Can you confirm if this is your setup and if that was intentional?

Thanks,
Kia

Kia-AWS
답변함 4년 전
0

Thanks for your response! I can confirm that this is my setup but it was not intentional. This configuration was set up several months ago and we haven't really touched it since then, besides adding a few more listeners to the load balancer itself. Typically, codedeploy will be responsible for shifting the traffic between blue and green and i've never manually adjusted it, i'm not even sure how i'd do that without codedeploy! Any solutions to this would be greatly appreciated, thank you!

kenndel
답변함 4년 전
0

Hi kenndel

Thank you for letting me know, the following link can help walk you through Load Balancer Set Up, Target Groups, and Listeners for blue/green ECS Deployments:

https://docs.aws.amazon.com/codedeploy/latest/userguide/deployment-groups-create-load-balancer-for-ecs.html

Let me know if you still need, I am happy to setup a call to walk you through the changes.

Thanks
Kia

Kia-AWS
답변함 4년 전
0

Hi Kia, I've reviewed those docs again and i don't think they help my situation. I also have a support ticket in for this issue but I'd love to hop on a call to discuss the fix!

kenndel
답변함 4년 전
0

Please check your Private Messages so I can connect with you directly and help you resolve your configuration.

Kia-AWS
답변함 4년 전
0

This issue has been resolved via a support ticket. I'll be posting the resolution here if anyone comes across it. The problem was, at some point during a deployment, traffic was moved 100% to my Green TG, my Blue TG was set to 0% traffic, but Blue TG was not removed from the load balancer rule. As a result, when i began a new deployment it noticed i had rules for both TGs and backed down. Pretty simple fix but, as i've never looked at the LB for this reason since getting Blue/Green deployments set up i never noticed that the blue TG at 0% was actually a problem, i assumed it just scaled them back and forth from 0 to 100 as needed. I hope this helps someone in the future, here are the steps:

Next Steps

As a workaround, please consider to make modifications on load balancer listener rules by following the below steps.

1. Modify the load balancer listener rules "lb rule arn" and "other lb rule arn" to point to **target group "green tg name" only and also delete the weight value for target group.**  

2. Modify the load balancer listener default rule "lb default rule arn" to point to target group "green tg name".  
  
3. Create a new deployment based on the same deployment group.  

Edited by: kenndel on Feb 11, 2020 12:43 PM

kenndel
답변함 4년 전
0

The ELB could not be updated due to the following error: Green taskset target group cannot have non-zero weight prior to traffic shifting on listener.
80 and 443 ports are listeners and the target groups both are forwarding to port number 80. listener 80 is forwarding to tg-2 and 443 is forwarding to tg-1 in ecs(blue/green) deployment.while updating with new revision number in ecs the revision number is not updating .the deploy stays in blue task set only not moving to green one

prathap
답변함 3년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠