Outbound Ports 80 and 443 being blocked from instance

Question

So, this has been keeping me busy for the past couple of days.  Started when I was troubleshooting the Paypal integration -- which is used only a couple of times a year when registration opens for an event.  It worked fine in October, but suddenly it stopped working.  I quickly figured out that the reason was that I couldn't connect to Paypal via port 443.  Upon further testing, I discovered I couldn't connect to *anything* on port 80 or 443.

Outbound SSH, FTP, and SMTP work fine from this instance.  I checked the ACLs for the VCP, which are allow any/any.  I checked my security group, which is also set to outbound any/any.  As a note, *inbound* HTTP and HTTPS both work just fine -- the website is still up.  Just that when I try to connect to anything else, even as root, it fails.

I have checked the configuration of the server, and there's nothing in iptables, and the Ubuntu firewall is disabled.  The server can connect to its own internal IP on port 80, but not its external IP.

I have another instance running, and on that instance I can connect to its internal IP on port 80, but not its external IP.

Reassociating the server with a different elastic IP gives the same behavior.

The other server can reach the Internet just fine on ports 80/443.

Things I have tried:
1. tcptraceroute fails immediately on the first hop.
2. All other ports that I have tried work fine.  Just 80 and 443 seem to be affected.
3. The behavior started sometime in the last 3 months.
4. tcpdump sees the SYN packets going outbound and supposedly leaving the interface.

So far, the only things I can think of that are consistent with the behavior:
1. The server has been compromised, or something got installed that is trying to capture/redirect all 80/443 traffic, but I can't think of anything or think where it would be.  It would have to be intercepted at the kernel level for tcpdump to see the SYN packets and think they are going out of eth0.  I'm not sure how to prove a negative here.  I may try creating a new instance using this server's volume and see what happens there.
2. Something associated with this particular instance is blocking outbound traffic, possibly upstream of us.  Does anyone know of any settings I haven't mentioned that would relate to this?

Any ideas are appreciated!

Answer

These symptoms would point to a Network ACL issue.

First, make sure that if you have multiple Network ACLs, you are looking at the one that is associated with the subnet your server is in.

Second, make sure you have (at least) 2 outbound & (at least) 2 inbound Network ACL rules - one of these will be the default ALL/DENY - you will also need a ALL/ALLOW with a destination of 0.0.0.0/0 for a default VPC configuration.

If you have other custom Network ACL rules, ensure they are in the correct order - they apply in order from the lowest rule number to the highest, with anything that doesn't match an explicit rule matching the default DENY rule (*).

Check the documentation for further information/examples: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html

If these look right - you may want to enabled VPC flow logs for this VPC to review exactly how far the traffic is getting, and any errors that might occur.

Answer

One really (probably bad) guess: is the instance configured to use a proxy server? Could it be trying to send the HTTP/S traffic there?

If that isn't it (which I doubt it): Raise a support case as the support team are able to look into the environment and determine if there is something in the VPC configuration which is blocking that traffic.

Outbound Ports 80 and 443 being blocked from instance

Relevant content