My Amazon Elastic Compute Cloud (Amazon EC2) instance average utilization is low, but the instance still exceeds its network limits.
Short description
You can query network performance metrics in real time on instances that support enhanced networking through the Elastic Network Adapter (ENA). These network performance metrics provide a cumulative value of the number of packets queued or dropped on each network interface since the last device reset.
The following are some of the ENA metrics:
- bw_in_allowance_exceeded: The number of packets queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance.
- bw_out_allowance_exceeded: The number of packets queued or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance.
- pps_allowance_exceeded: The number of packets queued or dropped because the bidirectional Packets Per Second (PPS) exceeded the maximum for the instance.
- conntrack_allowance_exceeded: The number of packets queued or dropped because the network traffic for the instance exceeded the maximum number of connections that can be tracked.
Network performance specifications vary according to the instance type. For more information on network performance specifications, see Amazon EC2 instance types. In some cases, you might see queuing or drops even though your average bandwidth or PPS as seen in Amazon CloudWatch is low. For example, the NetworkIn, NetworkOut, NetworkPacketsIn, or NetworkPacketsOut metrics in CloudWatch show amounts that don't suggest a limit being reached.
Resolution
Microbursts are the most common cause of the preceding symptoms. Microbursts are short spikes in demand followed by periods of low or no activity. Microbursts typically last only for seconds, milliseconds, or even microseconds. The CloudWatch metrics listed in the previous section aren't granular enough to reflect microbursts.
Average calculation
The EC2 metrics in CloudWatch listed in the previous section are sampled every 60 seconds. These metrics capture the total bytes or packets transferred in that period. These samples are then aggregated and published to CloudWatch in 5-minute periods. Each statistic in the period returns a different sample value:
- Sum is the sum of all sample values.
- SampleCount is the number of aggregated samples. In the previous example, there are 5 samples.
- Minimum is the sample value with the lowest byte or packet count.
- Average is the average sample value, calculated by dividing Sum by SampleCount.
- Maximum is the sample value with the highest byte or packet count.
You can calculate the average throughput or PPS in two ways:
- Divide Sum by Period. For example, divide the sum by 300 seconds for a simple 5-minute average.
- Divide Maximum by 60 seconds for an average in the minute with the highest activity.
Microbursts in CloudWatch metrics
The following is an example of a microburst, and how it appears in CloudWatch:
- The instance has a network bandwidth performance of 10 Gbps (1.25 GB/s).
- In a given sample of 60 seconds, an outbound data transfer of 20 GB uses up all available bandwidth. This action increments bw_out_allowance_exceeded. Transfer completes in about 20 seconds, and no further data is sent.
- Instance remains idle for the remaining 4 samples of 240 seconds.
In this example, the average throughput in a 5-minute period is much lower than the one during the microburst:
SUM(NetworkOut) / PERIOD = ((20 GB \* 1 sample) + (0 GB \* 4 samples)) / 300 seconds = ~0.066 GB/s \* 8 bits = ~0.533 Gbps
Even if you calculate the throughput based on the highest sample, the average still doesn't reflect the throughput amount:
MAXIMUM(NetworkOut) / 60 = 20 GB / 60 seconds = ~0.333 GB/s \* 8 bits = ~2.667 Gbps
Monitoring microbursts
To measure throughput and PPS at a more granular level, use operating system (OS) tools to monitor network statistics. Monitor your network statistics more frequently during periods of shaping or high activity.
The following are examples of OS tools that you can use:
Linux
- sar
- nload
- iftop
- iptraf-ng
Windows
The CloudWatch agent can be used on both Linux and Windows to publish OS-level network metrics to CloudWatch as custom metrics. These metrics can be published at intervals as low as 1 second. High-resolution metrics with a period less than 60 seconds lead to higher charges. For more information about CloudWatch pricing, see CloudWatch pricing.
It's a best practice to monitor the network performance metrics provided by ENA. To view the metrics, the driver version must be greater than or equal to 2.2.10 (Linux) and 2.2.2 (Windows. For more information, see Monitor network performance for your EC2 instance.
CloudWatch Agent can also publish ENA metrics. For instructions on publishing ENA metrics for Linux, see Collect network performance metrics. For Windows, ENA metrics are available in Performance Monitor. You can configure the CloudWatch Agent to publish available metrics in Performance Monitor.
Preventing microbursts
To avoid microbursts, traffic must be paced at the senders so that the traffic doesn't exceed a maximum throughput or packet rate. This makes microbursts difficult to avoid. To pace traffic at the senders, application-level changes are required. Depending on how this change is implemented, OS support for traffic pacing might need to be available and then turned on at the senders. However, that configuration might not always be possible or practical.
Microbursting can also happen because too many connections send packets in a short period. When this happens, pacing individual senders doesn't prevent this issue.
It's a best practice to monitor ENA metrics. If limits are reached often, or if there's evidence that your application is affected by traffic shaping, then do the following:
- Scale up: Move to a larger instance size. Larger instances have higher allowances. Network optimized instances, such as those with an "n" like C5n, have higher allowances than their non-network-optimized counterparts.
- Scale out: Spread traffic across multiple instances to reduce traffic and contention at individual instances.
For Linux-based operating systems, there are additional mitigation options for advanced users. You can implement the following options alone or in combination. It's a best practice to benchmark mitigations in a testing environment to verify that they reduce or eliminate traffic shaping without adverse effects on the workload.
SO_MAX_PACING_RATE
This socket option can be passed by an application to the setsockopt system call to specify a maximum pacing rate (bytes per second). The Linux kernel then paces traffic from that socket so that the traffic doesn't exceed the limit. This option requires the following:
- Application code-level changes.
- Support from the kernel. For more information, see net: introduce SO_MAX_PACING_RATE on the GitHub website.
- The use of Fair Queue (FQ) queuing discipline or the kernel's support for pacing at the TCP layer (applicable to TCP only).
For more information, see getsockopt(2) - Linux manual page and tc-fq(8) - Linux manual page on man7.org and tcp: internal implementation for pacing on the GitHub website.
Queuing disciplines (qdiscs)
qdiscs are responsible for packet scheduling and optional shaping. Some qdiscs, such as fq, help smooth out traffic bursts from individual flows. For more information, see the tc(8) - Linux manual page manual page on man7.org.
Shallow Transmission (Tx) queues
In some scenarios, shallow Tx queues help reduce PPS shaping. This reduction can be achieved in two ways:
- Byte Queue Limits (BQL): BQL dynamically limits the number of in-flight bytes on Tx queues. BQL is turned on by default on ENA driver versions shipped with the Linux kernel that ends with a "K". For ENA driver versions from GitHub that end with a "g", BQL is available in versions v2.6.1g and later, and is turned off by default. You can turn on BQL using the enable_bql ENA module parameter. For more information, see bql: Byte Queue Limits on lwn.net.
- Reduced Tx queue length: Reduce Tx queue length from its default of 1,024 packets to a lower amount (minimum 256). You can change this length using the ethtool. For more information, see ethtool(8) - Linux manual page on man7.org.
Related information
Amazon EC2 instance network bandwidth