I have a bunch of SQS services & s3 backup services that use a single IP address(NAT). As from today morning, I've lost complete connectivity to any & all aws services. Any TCP connection doesn't proceed beyond the firsy SYN packet. Has anyone ever heard of AWS perm-banning an IP address? I've got a bunch of business critical transactions stuck in SQS queues due to this :(
sudo tcpdump -i eth0 host 18.133.45.123 -n &
curl -v https://eu-west-2.queue.amazonaws.com/
* Trying 18.133.45.123...
* TCP_NODELAY set
16:20:47.610811 IP 197.248.216.154.33256 > 18.133.45.123.443: Flags [S], seq 2128825396, win 29200, options [mss 1460,sackOK,TS val 480045 ecr 0,nop,wscale 7], length 0
16:20:48.611248 IP 197.248.216.154.33256 > 18.133.45.123.443: Flags [S], seq 2128825396, win 29200, options [mss 1460,sackOK,TS val 480296 ecr 0,nop,wscale 7], length 0
16:20:50.627280 IP 197.248.216.154.33256 > 18.133.45.123.443: Flags [S], seq 2128825396, win 29200, options [mss 1460,sackOK,TS val 480800 ecr 0,nop,wscale 7], length 0
16:20:54.851253 IP 197.248.216.154.33256 > 18.133.45.123.443: Flags [S], seq 2128825396, win 29200, options [mss 1460,sackOK,TS val 481856 ecr 0,nop,wscale 7], length 0
16:21:01.934970 IP 197.248.216.154.42816 > 18.133.45.123.443: Flags [S], seq 3361955245, win 29200, options [mss 1460,sackOK,TS val 158275010 ecr 0,nop,wscale 7], length 0
16:21:02.960332 IP 197.248.216.154.42816 > 18.133.45.123.443: Flags [S], seq 3361955245, win 29200, options [mss 1460,sackOK,TS val 158275264 ecr 0,nop,wscale 7], length 0
16:21:03.043229 IP 197.248.216.154.33256 > 18.133.45.123.443: Flags [S], seq 2128825396, win 29200, options [mss 1460,sackOK,TS val 483904 ecr 0,nop,wscale 7], length 0
16:21:04.965428 IP 197.248.216.154.42816 > 18.133.45.123.443: Flags [S], seq 3361955245, win 29200, options [mss 1460,sackOK,TS val 158275768 ecr 0,nop,wscale 7], length 0
16:21:07.625705 IP 197.248.216.154.52394 > 18.133.45.123.443: Flags [S], seq 3840675465, win 29200, options [mss 1460,sackOK,TS val 3898989 ecr 0,nop,wscale 7], length 0
16:21:08.629690 IP 197.248.216.154.52394 > 18.133.45.123.443: Flags [S], seq 3840675465, win 29200, options [mss 1460,sackOK,TS val 3899240 ecr 0,nop,wscale 7], length 0
16:21:09.093703 IP 197.248.216.154.42816 > 18.133.45.123.443: Flags [S], seq 3361955245, win 29200, options [mss 1460,sackOK,TS val 158276800 ecr 0,nop,wscale 7], length 0
16:21:10.645819 IP 197.248.216.154.52394 > 18.133.45.123.443: Flags [S], seq 3840675465, win 29200, options [mss 1460,sackOK,TS val 3899744 ecr 0,nop,wscale 7], length 0
Console is not accessible too
sudo tcpdump -i eth0 host 99.83.252.222 -n &
curl -v http://console.aws.amazon.com/
* Trying 99.83.252.222...
* TCP_NODELAY set
16:21:46.099953 IP 197.248.216.154.36516 > 99.83.252.222.80: Flags [S], seq 773244091, win 29200, options [mss 1460,sackOK,TS val 494668 ecr 0,nop,wscale 7], length 0
16:21:47.107267 IP 197.248.216.154.36516 > 99.83.252.222.80: Flags [S], seq 773244091, win 29200, options [mss 1460,sackOK,TS val 494920 ecr 0,nop,wscale 7], length 0
16:21:49.123236 IP 197.248.216.154.36516 > 99.83.252.222.80: Flags [S], seq 773244091, win 29200, options [mss 1460,sackOK,TS val 495424 ecr 0,nop,wscale 7], length 0
16:21:53.219258 IP 197.248.216.154.36516 > 99.83.252.222.80: Flags [S], seq 773244091, win 29200, options [mss 1460,sackOK,TS val 496448 ecr 0,nop,wscale 7], length 0
As an aside, I have access to other devices (customers) from the same provider that are OK (i.e. it's not a provider routing issue/firewall)