I have a direct connect set up with multiple VPCs routed to/from the networks on the link. For the most part, the vast majority of traffic goes through just fine, however I have identified many intermittent cases where packets are just disappearing somewhere between our colo's direct connect interface port and the interface of the instance that the traffic is destined for.
I have used traffic mirroring to confirm that the packets are not reaching the ENI of the target instance, however there are many other packets destined for other systems in the same VPC in the same ~1ms window that are reaching those instances just fine. I have also identified other cases where other instances do not receive a packet, but the initial one does. It seems to be very random and not instance specific.
I have confirmed that our utilization is well below half of the circuit speed (~300mbps on a 1 gig), and the packets per second are relatively low overall (<20kpps), so I don't think it's a congestion issue on the link itself.
Can anyone point me in a direction to further troubleshoot this and identify where the packets are getting lost?
AWS OFFICIALUpdated 2 years ago
I have done all of the "usual" network diagnostics, the lost packets are very intermittent and very short lived so they don't show up on a traceroute or other utility that only checks once a second or more infrequently. I am only able to locate the issue reliably when reviewing the application logs of the sending device to see that it got a timeout, then confirming that the packet was sent but did not arrive in the packet captures I'm taking.
In that case working with the support is the best way forward.