ALB performing HTTPS-HTTP (SSL offloading), but the resulting HTTP request is gibberish

Question

First time I'm setting up a load balancer, and I must be missing something.

Created a new ALB with a certificate and listener for HTTPS.  It only has a single default rule for now, forwarding everything to a target group.  (It's an internal listener only, since during development we're all on a VPN.  If and when we ever want to migrate this to an internet-facing ALB, I'll probably set up more complicated rules with host headers to weed out the bots.)

The target group consists of one EC2 instance, specified with port 80 instead of 443.  The actual instance has some docker container running on it, because that's how that particular dev team rolls.  It's listening on port 80.

So the ALB / target group health checks are all passing.  (Getting the security groups right was a bit tricky; I'd forgotten that the SG on the ALB must also allow _outgoing_ traffic to originate, because that's where the health checks are coming from.)  And I can see the health checks in the instance's container logs:

```
$ sudo docker logs name-of-container
10.x.y.A - - [timestamp] "GET / HTTP/1.1" 200 687 "-" "ELB-HealthChecker/2.0"
10.x.y.B - - [timestamp] "GET / HTTP/1.1" 200 687 "-" "ELB-HealthChecker/2.0"
```

(Every health check shows up in pairs because the load balancer / target group has to be spread across at least 2 AZs, even if there's only a single instance actually found in those AZs.)

A request from a human with a web browser, inside the AWS address space going directly to that instance, shows up in the same logs as normal.  No surprises there.

But a request from a human with a web browser going to the load balancer shows up like this:

```
10.x.y.A - - [timestamp] "\x16\x03\x01\x00\xFE\x01\x00\x00 ......." 400 157 "-" "-"
```

which then causes the load balancer itself to return a 502; and the end user sees "502 Bad Gateway" in their browser.  And I know it's a human at a web browser, because it's often been **me triggering the request** while tail-ing the web server logs, it appears immediately.  The URL I entered was just the plain "https://cname.for.the.ALB.ourdomain.com/" which should have shown up as a `"GET /"` after the SSL offloading, but instead is... that.

I tried enabling access logs on the ALB itself, but there's nothing really obvious jumping out at me.  I see the 502 error, and the originating request `"GET https://internal-[our ALB name]-[numbers].us-east-2.elb.amazonaws.com:443/ HTTP/2.0"` along with the ARN of the target group, etc.

What in the world is the several dozen binary bytes being sent from the ALB and showing up in the log as hex?  As I said, each time corresponds to me trying to test with a real web browser and the most basic URL, so I don't expect the result to be some bizarre stack smashing attempt.  (If this is how I find out I have a split Manchurian Candidate personality, I will be embarrassed.)

re:Post Agent · Accepted Answer

The issue you're experiencing is likely due to a mismatch in the SSL/TLS handling between the Application Load Balancer (ALB) and your target instance. The gibberish you're seeing in the logs is actually encrypted HTTPS traffic being sent to your HTTP server, which is unable to process it correctly.

Here's what's happening:

1. Your ALB is configured to listen for HTTPS traffic on port 443.
2. The target group is set up to forward traffic to port 80 on your EC2 instance.
3. However, it appears that the ALB is not properly terminating the SSL connection and decrypting the traffic before forwarding it to your target.

As a result, the encrypted HTTPS data is being sent directly to your HTTP server, which interprets it as malformed HTTP requests, hence the gibberish in your logs.

To resolve this issue:

1. Double-check your ALB listener configuration. Ensure that the HTTPS listener is properly set up with the correct certificate.

2. Verify that your target group is configured correctly. It should be set to use the HTTP protocol (not HTTPS) when forwarding to your instances.

3. Make sure that the security policy on your HTTPS listener is compatible with your clients and properly configured for SSL termination.

4. If you're using any custom SSL policies or advanced features, review them to ensure they're not interfering with the SSL offloading process.

5. Check if there are any AWS Config rules or other security measures that might be affecting the SSL termination process.

If the issue persists after verifying these settings, you may want to try creating a new ALB and target group from scratch to see if the problem is resolved. Sometimes, recreating the resources can help if there's a hidden configuration issue.

Remember, when properly configured, the ALB should handle the SSL termination, decrypt the HTTPS traffic, and then forward plain HTTP requests to your target instances. This setup allows your instances to receive and process regular HTTP traffic while the ALB manages the SSL/TLS encryption and decryption.  
**Sources**  
[Create an HTTPS listener for your Application Load Balancer - Elastic Load Balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/create-https-listener.html)  
[Resolve HTTP connection errors with Application Load Balancer | AWS re:Post](https://repost.aws/knowledge-center/elb-alb-http-connection-errors)  
[Configure an HTTPS listener for your Classic Load Balancer - Elastic Load Balancing](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-add-or-delete-listeners.html)

ALB performing HTTPS->HTTP (SSL offloading), but the resulting HTTP request is gibberish

Relevant content