Can't access my lightsail system using aws load balancer. Health check 303.

Question

Hi all,

Problem description:

For testing purpose, I have a system with 2 lightsail instances connecting to a separated database. I set a port so that the each instance can be accessed via that port without any problem. Let's say the port is 1010. The 2 instances can be accessed using :1010 without any problem.

Now I want to access the whole lightsail system with a AWS load balancer via the 1010 port when connecting to one of them. And then I face a problem after doing the below steps:

1. I set up target group by using private IP of the 2 lightsail instances, with protocol = HTTP and Port = 1010
2. I set up a new security group to accept all traffic and all ip4 destination
3. In the "Listeners and routing" section, I add a record Protocol = HTTP, Port = 1010, target group = the created target group

But end up I fail to connect to my lightsail system. After some study, I found there is a error in my Target: Health Status = unhealthy, Health status details = Health checks failed with these codes: [303]

The weird part is individual server can be accessed very well, but not the case when connecting to load balancer.

My question is, are there any mistakes in my AWS settings causing this problem? If yes, hope I can get some hints to solve this.

A little bit long. Thank you.

Answer

it appears that the health check of the target group is failing, which is resulting in the "unhealthy" status. The health check failure code 303 suggests that the HTTP request to the target is being redirected to a different location.

Here are a few things you can check:

Health check settings: Make sure that the health check settings in the target group are configured correctly. Check the protocol, port, and path of the health check to ensure that they match the settings of your application.

Security group settings: Ensure that the security group of the instances allows traffic from the load balancer. Also, make sure that the security group of the load balancer allows traffic to the instances.

Application configuration: Verify that your application is configured to respond to requests from the load balancer. Ensure that the application is listening on the correct port and is not redirecting the requests to a different location.

Network settings: Check the network configuration of the instances to ensure that they can receive traffic from the load balancer. Make sure that the instances are not blocking traffic from the load balancer by using a firewall or other network security settings.

I hope these hints help you to identify and resolve the issue.

Answer

You'll need to enable VPC peering between your AWS account default VPC and Lightsail.

See:

I was able to create an application load balancer in my account that has a target group with "IP" target type. I needed to register IP addresses of my Lightsail instances using a command like this:

```
aws elbv2 register-targets --target-group-arn TARGET_GROUP_ARN \
   --targets Id=PRIVATE_IP1,AvailabilityZone=all Id=PRIVATE_IP2,AvailabilityZone=all
```

- The load balancer must be in your default VPC so that it can peer with the resources in the Lightsail VPC.
 - In each Lightsail instance's firewall settings, you will need to allow TCP traffic on port 1010 from your default VPC's CIDR (e.g. `172.31.0.0/16`,  but actual value of this CIDR can be found in VPC console for your default VPC) 
 - In the command above, you will need to specify `TARGET_GROUP_ARN`, `PRIVATE_IP1` and `PRIVATE_IP2` which are the private ip addresses of your two Lightsail instances.

More details can be found here:

Can't access my lightsail system using aws load balancer. Health check 303.

Relevant content