ELB Target group is not able to make the private subnet EC2 as healthy

Hello,

It’s normal for your Load Balancer to return a 504 error if it can’t properly reach its target group. Let’s troubleshoot step by step:

• Is your application actually running on the EC2 instance?
• Can you try accessing your application from another EC2 instance in the same subnet? Do you get a response?
• When you mention "allow all traffic," are you talking about both inbound and outbound rules?

Lastly, here’s something that has caught me off guard in the past: the Load Balancer expects a 200 status to consider the target healthy. If you’re using Nginx, it may be sending a 302 redirect instead. In this case, you need to configure the target group to treat it as a success. Here is how to do it:

1. Go to the EC2 console and click on Target Groups on the left panel.
2. Click on your target group
3. Click on the Health checks tab and edit it
4. Open the Advanced health check settings dropdown menu
5. Modify the Status codes field (e.g "200,302")

Hope this helps! Let me know if it does the trick 😊

Based on your description, there are several potential reasons why your ELB target group is not able to make the private subnet EC2 instance healthy, resulting in a 504 gateway timeout. Here are some steps to troubleshoot and potentially resolve the issue:

1. Network Configuration: Ensure that your VPC networking is set up correctly. The Load Balancer should be in a public subnet, while your EC2 instance can be in a private subnet. Make sure there's proper routing between these subnets.

2. Security Groups: Although you've allowed all traffic in both ELB and EC2 security groups, double-check that the security group associated with your EC2 instance allows inbound traffic on the Nginx port (likely 80 or 443) from the ELB's security group.

3. Target Group Health Checks: Verify that your target group's health check settings are correct. Ensure the health check path is valid and that Nginx is configured to respond to this path. Check if the health check port matches the port your application is running on.

4. Nginx Configuration: Confirm that Nginx is properly configured and running on your EC2 instance. Make sure it's listening on the correct port and interface (0.0.0.0 to listen on all interfaces, not just localhost).

5. EC2 Instance Network Interface: Check if the EC2 instance has the correct network interface attached and if it has a valid private IP address assigned.

6. VPC Peering: In a hub and spoke architecture, ensure that VPC peering is correctly set up between the hub and spoke VPCs, and that the necessary routes are in place in both VPCs' route tables.

7. Load Balancer Attributes: Check if cross-zone load balancing is enabled on your load balancer, which can help distribute traffic more evenly across instances in different Availability Zones.

8. Target Group Attributes: Review the target group attributes, particularly the deregistration delay and the healthy threshold count, to ensure they're appropriate for your setup.

If after checking these points you're still unable to resolve the issue, you may want to review the ELB access logs or CloudWatch metrics for more detailed information about the connection attempts and any potential issues. Additionally, you could try accessing the EC2 instance directly (via a bastion host if necessary) to ensure the application is running correctly.
Sources
Can't able to connect to the EC2 server in private network | AWS re:Post
Update health settings for your Application Load Balancer target group - Elastic Load Balancing
Target groups for your Application Load Balancers - Elastic Load Balancing