We have an inference pipeline reading an RTSP stream (GStreamer based) and performing inference (object detection/tracking). When we run our pipeline on our local server we have no issues and the system works as expected, however, when we move to deploy on AWS we suffer severe packet loss. Running tcpdump during testing on our local server we may recieve 1,000 warnings out of 300,000 packets, whereas on AWS EC2 (g4dn.xlarge) we recieve 100,000 warnings out of 300,000 packets. I analysed the traces in WireShark and on our server we see a repeating IO graph (60kb/s to 500kb/s) with minimal tcp errors, on AWS we see a repeating IO graph (60kb/s -> 700kb/s) with several tcp errors. To give a bit more insight we also measure camera read time to be about 1ms or less on our local server, on AWS we see ranges from 20ms to 120ms for camera read time.
Does anyone have any tips/things we can try to figure out what's going on?
Thanks!
