EC2 VM doesn't have Internet connection

0

I have a testing VPC created long time ago, it's connected to our On-Prem network with a VPN connection. We don't assign public IP in EC2 instances in this VPC and only access them through the VPN.

It was working before, but today when I powered up the only EC2 instance in the VPC, I found it lost Internet access. I can still ssh into it with the private IP from On-Prem, but it can't reach any website by curl, I can't ping any public IP including 8.8.8.8 ( DNS resolving is still working ).

[root@ip-10-240-46-19 ec2-user]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

[root@ip-10-240-46-19 ec2-user]# nslookup

www.yahoo.com
Server: 10.240.46.2
Address: 10.240.46.2#53

Non-authoritative answer:
www.yahoo.com canonical name = atsv2-fp-shed.wg1.b.yahoo.com.
Name: atsv2-fp-shed.wg1.b.yahoo.com
Address: 98.137.246.8

[root@ip-10-240-46-19 ec2-user]# ping www.yahoo.com
PING atsv2-fp-shed.wg1.b.yahoo.com (98.137.246.8) 56(84) bytes of data.
^C
--- atsv2-fp-shed.wg1.b.yahoo.com ping statistics ---

[root@ip-10-240-46-19 ec2-user]# yum update
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Could not retrieve mirrorlist http://amazonlinux.us-west-1.amazonaws.com/2/core/latest/x86_64/mirror.list error was
12: Timeout on http://amazonlinux.us-west-1.amazonaws.com/2/core/latest/x86_64/mirror.list: (28, 'Connection timed out after 5000 milliseconds')

[root@ip-10-240-46-19 ec2-user]# curl -v http://amazonlinux.us-west-1.amazonaws.com/2/core/latest/x86_64/mirror.list

  • Trying 52.219.120.48...

  • TCP_NODELAY set

  • Trying 2600:1fa0:c040:250:34db:78a1::...

  • TCP_NODELAY set

tcpdump shows there is no response from public server:
14:36:42.468736 IP 10.240.46.19.42110 > 52.219.24.33.80: Flags [S], seq 3488750344, win 26883, options [mss 8961,sackOK,TS val 1544269441 ecr 0,nop,wscale 7], length 0
14:36:43.475306 IP 10.240.46.19.42110 > 52.219.24.33.80: Flags [S], seq 3488750344, win 26883, options [mss 8961,sackOK,TS val 1544270448 ecr 0,nop,wscale 7], length 0
14:36:45.491327 IP 10.240.46.19.42110 > 52.219.24.33.80: Flags [S], seq 3488750344, win 26883, options [mss 8961,sackOK,TS val 1544272464 ecr 0,nop,wscale 7], length 0

I did some troubleshooting:

  1. checked security group:
    There is only one inbound rule to allow TCP 22; one outbound rule to allow "All All 0.0.0.0/0".

  2. Checked VPC routing table, there are three rules:
    10.240.46.0/24 local 0.0.0.0/0 -> internet gateway
    10.0.0.0/8 -> VPN Gateway ( apparent this one still working )

  3. Tried to create a new Internet Gateway and swap it, didn't fix it.

  4. Tried to create a Nat gateway and changed routing 0.0.0.0/0 to use the Nat Gateway, didn't fix it.

  5. removed Nat Gateway, changed it back to use Internet Gateway for 0.0.0.0/0.

  6. Created a new EC2 instance in same subnet with a different private IP, same problem.

Inside the instance, it shows:
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.240.46.1 0.0.0.0 UG 0 0 0 eth0
10.240.46.0 0.0.0.0 255.255.255.224 U 0 0 0 eth0
169.254.169.254 0.0.0.0 255.255.255.255 UH 0 0 0 eth0

Now I run out of idea.
Is there some where else I should be looking?

Thanks,
Jack

asked 5 years ago673 views
3 Answers
0

forgot to mention, also checked VPC ACL,its outbound rule has rule 100 to allow any.

answered 5 years ago
0

OK will answer the question myself:

opened support ticket with AWS and the support answered my question:
"In order for your NAT Gateway to work properly, you will have to place it in a public subnet ie. the subnet should have an internet gateway attached to it and you should have subnet specific route tables where the 0.0.0.0/0 on your private subnet is routed to your NAT Gateway and the 0.0.0.0/0 on your public subnet is routed to the Internet Gateway."

https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html

Worked mostly in Azure ( where Internet gateway and NAT gateway were setup directly by Azure Vnet ), I didn't realize how private network should be configured in AWS.

answered 4 years ago
0

Hi jackchen858,

I'm glad that you were able to solve the issue. Please let us know if you need any further help.

Regards,
awstomas

AWS
answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions