Network traffic limits on x2gd.medium

Question

We have setup redis on x2gd.medium with 2 to 5 ec2 instances connected within the same region
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/memory-optimized-instances.html indicates it has a baseline of .5Gbps.
From the network In/Out monitoring, it seems we are averaging about .35Gbps with some minor peaks.
https://photos.app.goo.gl/rkizVakGQpZo8v5q6

Yet we are observing constant increasing of 
```
     ethtool -S ens5
     bw_in_allowance_exceeded: 314
     bw_out_allowance_exceeded: 281794
```

Our setup is fairly vanilla
- ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20220419
ami-0d70a59d7191a8079
- add-apt-repository -y ppa:redislabs/redis
- apt-get -y update
- apt-get install -y nvme-cli
- apt-get -y install redis

```
uname -a
Linux ip-10-0-0-41 5.13.0-1022-aws #24~20.04.1-Ubuntu SMP Thu Apr 7 22:14:11 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

redis-cli -v
redis-cli 7.0.0

modinfo ena
filename:       /lib/modules/5.13.0-1022-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko
license:        GPL
description:    Elastic Network Adapter (ENA)
author:         Amazon.com, Inc. or its affiliates
srcversion:     066F20794698BA58475C910
alias:          pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000051sv*sd*bc*sc*i*
depends:
intree:         Y
name:           ena
vermagic:       5.13.0-1022-aws SMP mod_unload modversions aarch64
```

Question:
1. How can we know more about this "hidden networking credits" that seems to be limiting our bandwidth to avoid exceeding the allowance?

Accepted Answer

Hello Freedom_AWS,

To piggyback off of GaryKo_AWS, the most likely reason for the allowance exceeded increments and under baseline utilization is due to microbursting. An example of this would be if there is a microburst lasting 10 seconds that is above the BW allowance, then there are no more microbursts and utilization under .5 gbps for 4 minutes and 50 seconds. The CloudWatch metric period of 5 minutes will show the lower utilization and will not reflect the microburst due to CW not being granular enough while still incrementing the bw_out_allowance_exceeded. To monitor throughput / PPS at a more granular level and get more visibility into what is happening, use OS tools to monitor network statistics.

[Here is a more in depth guide about it.](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-exceeding-network-limits/)

Answer

Following up on this, we have split the traffic to two r6gd.medium server with .5Gbps each, which unfortunately we are still seeing continue increasing of 
```
bw_out_allowance_exceeded
```

This is really frustrating where we are not able to find/configure an instance that we won't be overcharged for stuff that we won't be utilizing.

Answer

An instance receives the maximum number of network I/O credits at launch. That is the number (Burst bandwidth) you can find from the userguide. When the instance exhausts its network I/O credits, it returns to its baseline bandwidth. A running instance earns network I/O credits whenever it uses less network bandwidth than its baseline bandwidth. A stopped instance does not earn network I/O credits. Instance burst is on a best effort basis, even when the instance has credits available, as burst bandwidth is a shared resource. (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html)

In order to achieve higher baseline network bandwidth, I would recommend you change your instance size, and perform regularly benchmark on the bandwidth between instances using tools like iperf.

In order to achieve higher baseline network bandwidth, I would recommend you change your instance size, and perform regularly benchmark on the bandwidth between instances using tools like  **iperf**.

Network traffic limits on x2gd.medium

Relevant content