Network traffic limits on x2gd.medium
We have setup redis on x2gd.medium with 2 to 5 ec2 instances connected within the same region https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/memory-optimized-instances.html indicates it has a baseline of .5Gbps. From the network In/Out monitoring, it seems we are averaging about .35Gbps with some minor peaks. https://photos.app.goo.gl/rkizVakGQpZo8v5q6
Yet we are observing constant increasing of
ethtool -S ens5 bw_in_allowance_exceeded: 314 bw_out_allowance_exceeded: 281794
Our setup is fairly vanilla
- add-apt-repository -y ppa:redislabs/redis
- apt-get -y update
- apt-get install -y nvme-cli
- apt-get -y install redis
uname -a Linux ip-10-0-0-41 5.13.0-1022-aws #24~20.04.1-Ubuntu SMP Thu Apr 7 22:14:11 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux redis-cli -v redis-cli 7.0.0 modinfo ena filename: /lib/modules/5.13.0-1022-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko license: GPL description: Elastic Network Adapter (ENA) author: Amazon.com, Inc. or its affiliates srcversion: 066F20794698BA58475C910 alias: pci:v00001D0Fd0000EC21sv*sd*bc*sc*i* alias: pci:v00001D0Fd0000EC20sv*sd*bc*sc*i* alias: pci:v00001D0Fd00001EC2sv*sd*bc*sc*i* alias: pci:v00001D0Fd00000EC2sv*sd*bc*sc*i* alias: pci:v00001D0Fd00000051sv*sd*bc*sc*i* depends: intree: Y name: ena vermagic: 5.13.0-1022-aws SMP mod_unload modversions aarch64
- How can we know more about this "hidden networking credits" that seems to be limiting our bandwidth to avoid exceeding the allowance?
To piggyback off of GaryKo_AWS, the most likely reason for the allowance exceeded increments and under baseline utilization is due to microbursting. An example of this would be if there is a microburst lasting 10 seconds that is above the BW allowance, then there are no more microbursts and utilization under .5 gbps for 4 minutes and 50 seconds. The CloudWatch metric period of 5 minutes will show the lower utilization and will not reflect the microburst due to CW not being granular enough while still incrementing the bw_out_allowance_exceeded. To monitor throughput / PPS at a more granular level and get more visibility into what is happening, use OS tools to monitor network statistics.
Thanks for providing the link, not sure why the link doesn't show up on search. I also found this https://www.bluematador.com/blog/how-many-packets-per-second-in-amazon-ec2 confirming the spiky and unpredictable nature when bandwidth is exceeded.
As it'll be unlikely that we'll be able to change the app itself, scaling up/out seems like the only way to go.
An instance receives the maximum number of network I/O credits at launch. That is the number (Burst bandwidth) you can find from the userguide. When the instance exhausts its network I/O credits, it returns to its baseline bandwidth. A running instance earns network I/O credits whenever it uses less network bandwidth than its baseline bandwidth. A stopped instance does not earn network I/O credits. Instance burst is on a best effort basis, even when the instance has credits available, as burst bandwidth is a shared resource. (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html)
In order to achieve higher baseline network bandwidth, I would recommend you change your instance size, and perform regularly benchmark on the bandwidth between instances using tools like iperf.
We got the same info from the AWS documents. However, it doesn't make much sense that we are under the .5Gbps baseline for the past 3 days, yet we are still seeing increasing allowance_exceeded
Wed Jun 8 08:48:52 UTC 2022 bw_in_allowance_exceeded: 330 bw_out_allowance_exceeded: 325692
Following up on this, we have split the traffic to two r6gd.medium server with .5Gbps each, which unfortunately we are still seeing continue increasing of
This is really frustrating where we are not able to find/configure an instance that we won't be overcharged for stuff that we won't be utilizing.
Improving copying speed across EC2 instancesasked 3 months ago
Network traffic limits on EC2 instancesasked 5 months ago
How to get EC2 instance to access internet facing Network Loadbalancer.asked a year ago
Routing network traffic between two EC2 instances in the same subnet to a firewall appliance in another VPCAccepted Answerasked 5 months ago
Connection is filtered when using Elastic IP for EC2 Linux 2asked 2 years ago
If a Lambda @ Edge function calls DynamoDB, does the traffic stay within the AWS network?Accepted Answerasked a year ago
Where does outbound traffic exit the Amazon network?Accepted Answerasked 2 years ago
We need to point two domains to the same EC2. It is not working.asked 12 days ago
Network traffic limits on x2gd.mediumAccepted Answerasked a month ago
Memory and CPU allocation for EC2 on Free Tier.asked 4 months ago