Removing Identical Security Group from ALB causes 504 errors

Question

Hi,

I have an ALB which I'm trying to replace the security group with a security group with the same rules (for purposes of naming). Both security groups are attached to the ALB.

When I remove the old Security group, my application reports a number of 504 errors. Is this expected behaviour?

Riku_Kobayashi · Accepted Answer

Hello.

Please check the inbound rules of the security groups used by the ALB targets, such as EC2 or ECS.  
This is just a guess, but I think that the older security group is allowed in the ALB target security group, but the newer security group is not.  
If the inbound rules of a security group reference another security group, you must also change the referencing security group.  
https://docs.aws.amazon.com/vpc/latest/userguide/security-group-rules.html#security-group-referencing

re:Post Agent · Answer

The behavior you're experiencing, where removing a security group from an Application Load Balancer (ALB) causes 504 errors, is not typically expected if the new security group has identical rules to the old one. However, there are a few factors to consider:

1. Propagation delay: When you make changes to security groups, there can be a brief delay as the changes propagate across the AWS network. During this time, you might experience some temporary connectivity issues.

2. Connection draining: Existing connections may be affected when you change security groups, potentially leading to some requests timing out.

3. Unintended differences: Double-check that the new security group truly has identical rules to the old one. Even small differences could cause connectivity issues.

4. Backend issues: 504 Gateway Timeout errors often indicate that the ALB couldn't establish a connection with the backend targets. Ensure that your backend instances or services are healthy and responding properly.

To troubleshoot this issue:

1. Verify that both security groups have identical rules.
2. Check the health of your target instances or services.
3. Monitor your ALB logs and metrics during the security group change.
4. Consider making the change during a low-traffic period.
5. If possible, add the new security group first, then remove the old one after a short delay to allow for propagation.

If the issue persists even with these precautions, it may be worth engaging AWS support to investigate further, as this behavior is not typical when swapping identical security groups.  
**Sources**  
[ALB Security Group | AWS re:Post](https://repost.aws/questions/QUmNx6Q10pSMuvy9cz--D_hg/alb-security-group)  
[Resolve HTTP connection errors with Application Load Balancer | AWS re:Post](https://repost.aws/knowledge-center/elb-alb-http-connection-errors)  
[Application Load Balancer 504 errors with weighted target group | AWS re:Post](https://repost.aws/questions/QU83vc4Oc-QPCu04XEAOzN7Q/application-load-balancer-504-errors-with-weighted-target-group)

Removing Identical Security Group from ALB causes 504 errors

Relevant content