By using AWS re:Post, you agree to the Terms of Use

Questions tagged with Networking & Content Delivery

Sort by most recent
  • 1
  • 12 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

EC2 outbound bandwidth dropped on July 21st and stays very low (eu-west-1)

Hello, In our organization, we run our CICD using gitlab. We spawn runners on demand and each runner is a EC2 instance. Since July 21st, the uploads performed by these runners got their duration multiplied by ~10. From our tests, we see that: - outbound bandwidth from any instance type and any AMI we can spawn in eu-west-1 is now <2,5Mb/s (~300KB/s), - we tested t3.medium and m6a.large, - we tested various AMIs (ubuntu, amazon linux), - this seems far from any quota/limit/advertised bandwidth, - before July 21st, it was around 25Mb/s (~3MB/s) for t3.medium runners, - download speed is 500Mb/s, - upload/download target is the gitlab server host itself, - located in Paris, - can be reached through 2 different physical connections and we get very good speed from other places, - all speed tests mentioned here are performed using iperf3, - runners are in a VPC, routed through an Internet Gateway, - if we spawn a runner in eu-west-3, outbound speed is 24Mb/s (and download is 1Gb/s), - that's not great, but at least it’s 10 times what we get in eu-west-1 (and close to what we had before “the incident”). - we haven’t changed a thing regarding our configuration, runners, gitlab version, code. The problem happened suddenly for pipelines run after July 21st, 4pm cest, We may consider spawning our runners in eu-west-3, or even move our gitlab server on EC2, but since 1) the problem happened suddenly without any action on our side and 2) this would require moving other resources as well, time, effort and cost, we’d rather prefer a logic explanation of what’s happening here before taking actions. We are looking for clues to understand: - how to monitor this in order to identify if we actually reach a limit/quota, - why this changed suddenly, - why outbound bandwidth is that low (is that to be expected, or not). Any help investigating would be greatly appreciated! Best,
0
answers
0
votes
27
views
asked 6 days ago
  • 1
  • 12 / page