Network traffic limits on x2gd.medium

Question

We have setup redis on x2gd.medium with 2 to 5 ec2 instances connected within the same region
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/memory-optimized-instances.html indicates it has a baseline of .5Gbps.
From the network In/Out monitoring, it seems we are averaging about .35Gbps with some minor peaks.
https://photos.app.goo.gl/rkizVakGQpZo8v5q6

Yet we are observing constant increasing of 
```
     ethtool -S ens5
     bw_in_allowance_exceeded: 314
     bw_out_allowance_exceeded: 281794
```

Our setup is fairly vanilla
- ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20220419
ami-0d70a59d7191a8079
- add-apt-repository -y ppa:redislabs/redis
- apt-get -y update
- apt-get install -y nvme-cli
- apt-get -y install redis

```
uname -a
Linux ip-10-0-0-41 5.13.0-1022-aws #24~20.04.1-Ubuntu SMP Thu Apr 7 22:14:11 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

redis-cli -v
redis-cli 7.0.0

modinfo ena
filename:       /lib/modules/5.13.0-1022-aws/kernel/drivers/net/ethernet/amazon/ena/ena.ko
license:        GPL
description:    Elastic Network Adapter (ENA)
author:         Amazon.com, Inc. or its affiliates
srcversion:     066F20794698BA58475C910
alias:          pci:v00001D0Fd0000EC21sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd0000EC20sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00001EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000EC2sv*sd*bc*sc*i*
alias:          pci:v00001D0Fd00000051sv*sd*bc*sc*i*
depends:
intree:         Y
name:           ena
vermagic:       5.13.0-1022-aws SMP mod_unload modversions aarch64
```

Question:
1. How can we know more about this "hidden networking credits" that seems to be limiting our bandwidth to avoid exceeding the allowance?

Accepted Answer

Hello Freedom_AWS,

To piggyback off of GaryKo_AWS, the most likely reason for the allowance exceeded increments and under baseline utilization is due to microbursting. An example of this would be if there is a microburst lasting 10 seconds that is above the BW allowance, then there are no more microbursts and utilization under .5 gbps for 4 minutes and 50 seconds. The CloudWatch metric period of 5 minutes will show the lower utilization and will not reflect the microburst due to CW not being granular enough while still incrementing the bw_out_allowance_exceeded. To monitor throughput / PPS at a more granular level and get more visibility into what is happening, use OS tools to monitor network statistics.

[Here is a more in depth guide about it.](https://aws.amazon.com/premiumsupport/knowledge-center/ec2-instance-exceeding-network-limits/)

Answer

Following up on this, we have split the traffic to two r6gd.medium server with .5Gbps each, which unfortunately we are still seeing continue increasing of 
```
bw_out_allowance_exceeded
```

This is really frustrating where we are not able to find/configure an instance that we won't be overcharged for stuff that we won't be utilizing.

Answer

An instance receives the maximum number of network I/O credits at launch. That is the number (Burst bandwidth) you can find from the userguide. When the instance exhausts its network I/O credits, it returns to its baseline bandwidth. A running instance earns network I/O credits whenever it uses less network bandwidth than its baseline bandwidth. A stopped instance does not earn network I/O credits. Instance burst is on a best effort basis, even when the instance has credits available, as burst bandwidth is a shared resource. (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-network-bandwidth.html)

In order to achieve higher baseline network bandwidth, I would recommend you change your instance size, and perform regularly benchmark on the bandwidth between instances using tools like iperf.

In order to achieve higher baseline network bandwidth, I would recommend you change your instance size, and perform regularly benchmark on the bandwidth between instances using tools like  **iperf**.

Network traffic limits on x2gd.medium

Relevant questions

Improving copying speed across EC2 instances

Network traffic limits on EC2 instances

How to get EC2 instance to access internet facing Network Loadbalancer.

Routing network traffic between two EC2 instances in the same subnet to a firewall appliance in another VPC

Connection is filtered when using Elastic IP for EC2 Linux 2

If a Lambda @ Edge function calls DynamoDB, does the traffic stay within the AWS network?

Where does outbound traffic exit the Amazon network?

We need to point two domains to the same EC2. It is not working.

Network traffic limits on x2gd.medium

Memory and CPU allocation for EC2 on Free Tier.