A couple of weeks ago my site-to-site VPN stopped functioning. After troubleshooting I opted to rebuild it and I'm still encountering the same issue.
My VPN itself comes up, and if I ping from on-premise to my EC2 instance over the tunnels the traffic makes it to the VM (based on packet captures) The VM also sends back an icmp response - but no response traffic is never received.
[root@dns0 ~]# tcpdump -i any icmp -n tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes 17:11:29.066164 IP 192.168.32.15 > 10.2.0.100: ICMP echo request, id 6701, seq 1, length 64 17:11:29.066197 IP 10.2.0.100 > 192.168.32.15: ICMP echo reply, id 6701, seq 1, length 64 17:11:30.068755 IP 192.168.32.15 > 10.2.0.100: ICMP echo request, id 6701, seq 2, length 64 17:11:30.068786 IP 10.2.0.100 > 192.168.32.15: ICMP echo reply, id 6701, seq 2, length 64
I've performed packet captures also on the router upstream from my VPN appliance (which Fortigate), and the router is forwarding all packets it's receiving. So it looks like something is blocking ESP traffic from AWS to me.
flamethrowerX # diagnose vpn tunnel list name AWS_VPN_Tun1
list ipsec tunnel by names in vd 0
name=AWS_VPN_Tun1 ver=1 serial=b 192.168.70.2:4500->35.161.211.159:4500 dst_mtu=1500
bound_if=5 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/536 options[0218]=npu create_dev frag-rfc accept_traffic=1 overlay_id=0
proxyid_num=1 child_num=0 refcnt=15 ilast=2 olast=2 ad=/0
stat: rxp=0 txp=137 rxb=0 txb=13118
dpd: mode=on-demand on=1 idle=10000ms retry=3 count=0 seqno=1961
natt: mode=keepalive draft=32 interval=10 remote_port=4500
proxyid=AWS_VPN_Tun1 proto=0 sa=1 ref=3 serial=6 auto-negotiate
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
SA: ref=5 options=18227 type=00 soft=0 mtu=1422 expire=2574/0B replaywin=2048
seqno=88 esn=0 replaywin_lastseq=00000000 itn=0 qat=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=3298/3600
dec: spi=425a25f9 esp=aes key=16 272182d3d84386f288619faec7f50877
ah=sha1 key=20 a9cc129aaa662c965aa37458a9180478d4377a68
enc: spi=c8f7851d esp=aes key=16 b52b69c938f1fe14f3bf5bd015c3cf77
ah=sha1 key=20 2c55bb134da8ad5b79080e1efe06c8d742fa5cc3
** dec:pkts/bytes=0/0, enc:pkts/bytes=137/13574**
npu_flag=01 npu_rgwy=35.161.211.159 npu_lgwy=192.168.70.2 npu_selid=15 dec_npuid=0 enc_npuid=1
flamethrowerX # diagnose vpn tunnel list name AWS_VPN_Tun2
list ipsec tunnel by names in vd 0
name=AWS_VPN_Tun2 ver=1 serial=c 192.168.70.2:4500->52.34.101.205:4500 dst_mtu=1500
bound_if=5 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/536 options[0218]=npu create_dev frag-rfc accept_traffic=1 overlay_id=0
proxyid_num=1 child_num=0 refcnt=13 ilast=6 olast=547 ad=/0
stat: rxp=0 txp=0 rxb=0 txb=0
dpd: mode=on-demand on=1 idle=10000ms retry=3 count=0 seqno=2036
natt: mode=keepalive draft=32 interval=10 remote_port=4500
proxyid=AWS_VPN_Tun2 proto=0 sa=1 ref=2 serial=5 auto-negotiate
src: 0:0.0.0.0/0.0.0.0:0
dst: 0:0.0.0.0/0.0.0.0:0
SA: ref=3 options=18227 type=00 soft=0 mtu=1422 expire=2752/0B replaywin=2048
seqno=1 esn=0 replaywin_lastseq=00000000 itn=0 qat=0 hash_search_len=1
life: type=01 bytes=0/0 timeout=3299/3600
dec: spi=425a2600 esp=aes key=16 4befd18314654756c8325097e578afbf
ah=sha1 key=20 42a9c5acfeb2359f3ab638203c765f5966bdb211
enc: spi=c335e109 esp=aes key=16 a3173f068a7ac0fcfff39b26c39bf180
ah=sha1 key=20 5b797b43d59e7d8929bd5d3f644f10a51bf414a1
** dec:pkts/bytes=0/0, enc:pkts/bytes=0/0**
npu_flag=00 npu_rgwy=52.34.101.205 npu_lgwy=192.168.70.2 npu_selid=16 dec_npuid=0 enc_npuid=0
Routes in AWS are fine - they're correctly pointing to the VGW.
NACLs are good.
Security Groups are configured correctly.
Is there anything else I should look at? I'm at a total loss to understand why this suddenly broke.
