Possible network insfrastructure corrput? Can't access port 80 in a given instance.

Based on your description, it sounds like you're experiencing an issue where ports 80 and 443 are not accessible on your EC2 instance, despite security groups being properly configured. Since you can access these ports locally from the server itself and a new instance in the same VPC with the same security group works fine, the issue is likely at the operating system level rather than with AWS networking infrastructure.

Here are some troubleshooting steps to consider:

1. Check if your operating system's firewall is blocking these ports. Depending on your OS:

• For Ubuntu: Check UFW status with sudo ufw status numbered and if it's active, allow access with sudo ufw allow 80 and sudo ufw allow 443
• For RHEL/Rocky/Alma Linux: Check if firewalld is blocking traffic with sudo systemctl status firewalld

2. Verify that your web server application (Apache, Nginx, etc.) is running properly:

• Check if the service is active using sudo netstat -tlnp to see if it's listening on ports 80/443
• Restart the web server service

3. Check network ACLs to ensure they're not blocking traffic to these ports

4. Verify route tables are correctly configured to allow outbound and inbound traffic

If none of these steps resolve the issue, you might consider creating an AMI (Amazon Machine Image) of your current instance and launching a new instance from that AMI. This would effectively "migrate" your data to a new instance that might not have the same underlying issue.

For immediate business continuity, you could also consider setting up a temporary solution by configuring your services to run on different ports that are working (like your 7000-7999 range) until the root cause is identified.
Sources
Troubleshoot connection errors | AWS re:Post
Unable to Connect Http ports Even after allowing in Security group. | AWS re:Post
Troubleshoot my failed Amazon ECS deployments | AWS re:Post
Behavior of ec2 on port 8080 | AWS re:Post

It's not in the OS level.

1. local firewall:

ubuntu@aws-hellotrip-docker:~$ sudo ufw status numbered Status: inactive

2. web application:

The web application was completely stopped and I was testing with nc -l $PORT When $PORT was 7666, doing a telnet $MY_IP 7666 worked. When $PORT was 80, doing a telnet $MY_IP 80, it did not connect.

3. ACLs

The corresponding ACL has only one rule that accepts all:

100 | All traffic | All | All | 0.0.0.0/0 | Allow

• | All traffic | All | All | 0.0.0.0/0 | Deny

4. Routing

0.0.0.0/0 | igw-85nnnnnn | Active | No 172.30.0.0/16 | local | Active | No

All the traffic is routed to igw-*

Can't I request Amazon to "move" my machine "as is"? I think they do that from time to time when something fails. This would avoid creating the AMI, launching, testing, etc.

Okay! So I solved it.

This article "Change the instance type for your Amazon EC2 instance" https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/change-instance-type-of-ebs-backed-instance.html is about changing the instance type.

Back in 2012 when you created an instance, it was that specific type forever, but AWS does a great job at virtualizong more and more abstraction levels so now you can upgrade/downgrade the instance type. This might require a hardware change.

The article says:

" Considerations

• [...]
• When you stop and start an instance, we move the instance to new hardware. [...] "

So... what if you stop, do NOT change the instance type and start again?

This is what I did.

During all my tests, I rebooted the machine via command line "sudo reboot", but I assume that's pure software and from the AWS infrastructure that's a non-interrupted working machine. And the problem did NOT go away.

Instead, I stopped the machine (I have an elastic IP address attached and a couple of EBS volumes). I waited 2 or 3 minutes. Then I started the machine again. I don't know if it truly got running on top of a new hardware or not, but what I can say is that the port problem was hapily resolved without having to fully configure a new fresh instance!

Unfortunately, it worked only for some minutes. After 3 to 8 mins, the ports were blocked again. So I tried to discover what hardware it was running on.

Here "Detect whether a host is an EC2 instance" https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html it says that doing this:

cat /sys/hypervisor/uuid

we get the underlying infrastructure Id.

Unfortunately I got ec2e1603-7d1f-ee17-51ec-b77bcbc59e39, then I stopped, I waited, I started and got the very same ID after starting again.

I then forced a type change. Moved from t2.micro to t3.micro.

I got the nitro supervisor and doing this cat /sys/devices/virtual/dmi/id/board_asset_tag I got this: i-05e21fec368f0f181

Then I stopped again and moved again back to t2,micro hoping the underlying hardware would be another one. But unfortunately II got the very same ec2e1603-7d1f-ee17-51ec-b77bcbc59e39 and the problem with the ports persisted: The very few minutes it works and after a few minutes it breaks again.

I think the only solution will be to create an AMI from it, and re-launch a fresh instance.

After 1 full night with the machine stopped, with the expectation that today the /sys/hypervisor/uuid changed upon start, it got onto the same hardware and the issue persisted.

I'm going to abandon the idea of resurrecting this instance and I'm going to install a fresh new server from scratch.