I have a testing VPC created long time ago, it's connected to our On-Prem network with a VPN connection. We don't assign public IP in EC2 instances in this VPC and only access them through the VPN.
It was working before, but today when I powered up the only EC2 instance in the VPC, I found it lost Internet access. I can still ssh into it with the private IP from On-Prem, but it can't reach any website by curl, I can't ping any public IP including 8.8.8.8 ( DNS resolving is still working ).
[root@ip-10-240-46-19 ec2-user]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
[root@ip-10-240-46-19 ec2-user]# nslookup
www.yahoo.com
Server: 10.240.46.2
Address: 10.240.46.2#53
Non-authoritative answer:
www.yahoo.com canonical name = atsv2-fp-shed.wg1.b.yahoo.com.
Name: atsv2-fp-shed.wg1.b.yahoo.com
Address: 98.137.246.8
[root@ip-10-240-46-19 ec2-user]# ping www.yahoo.com
PING atsv2-fp-shed.wg1.b.yahoo.com (98.137.246.8) 56(84) bytes of data.
^C
--- atsv2-fp-shed.wg1.b.yahoo.com ping statistics ---
[root@ip-10-240-46-19 ec2-user]# yum update
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
Could not retrieve mirrorlist http://amazonlinux.us-west-1.amazonaws.com/2/core/latest/x86_64/mirror.list error was
12: Timeout on http://amazonlinux.us-west-1.amazonaws.com/2/core/latest/x86_64/mirror.list: (28, 'Connection timed out after 5000 milliseconds')
[root@ip-10-240-46-19 ec2-user]# curl -v http://amazonlinux.us-west-1.amazonaws.com/2/core/latest/x86_64/mirror.list
tcpdump shows there is no response from public server:
14:36:42.468736 IP 10.240.46.19.42110 > 52.219.24.33.80: Flags [S], seq 3488750344, win 26883, options [mss 8961,sackOK,TS val 1544269441 ecr 0,nop,wscale 7], length 0
14:36:43.475306 IP 10.240.46.19.42110 > 52.219.24.33.80: Flags [S], seq 3488750344, win 26883, options [mss 8961,sackOK,TS val 1544270448 ecr 0,nop,wscale 7], length 0
14:36:45.491327 IP 10.240.46.19.42110 > 52.219.24.33.80: Flags [S], seq 3488750344, win 26883, options [mss 8961,sackOK,TS val 1544272464 ecr 0,nop,wscale 7], length 0
I did some troubleshooting:
-
checked security group:
There is only one inbound rule to allow TCP 22; one outbound rule to allow "All All 0.0.0.0/0".
-
Checked VPC routing table, there are three rules:
10.240.46.0/24 local
0.0.0.0/0 -> internet gateway
10.0.0.0/8 -> VPN Gateway ( apparent this one still working )
-
Tried to create a new Internet Gateway and swap it, didn't fix it.
-
Tried to create a Nat gateway and changed routing 0.0.0.0/0 to use the Nat Gateway, didn't fix it.
-
removed Nat Gateway, changed it back to use Internet Gateway for 0.0.0.0/0.
-
Created a new EC2 instance in same subnet with a different private IP, same problem.
Inside the instance, it shows:
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.240.46.1 0.0.0.0 UG 0 0 0 eth0
10.240.46.0 0.0.0.0 255.255.255.224 U 0 0 0 eth0
169.254.169.254 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
Now I run out of idea.
Is there some where else I should be looking?
Thanks,
Jack