Why does my Amazon EC2 instance exceed its network limits when average utilization is low?

7 minute read
1

My Amazon Elastic Compute Cloud (Amazon EC2) instance average utilization is low, but the instance still exceeds its network limits.

Short description

You can query network performance metrics in real time on instances that support enhanced networking through the Elastic Network Adapter (ENA). These network performance metrics provide a cumulative value of the number of packets queued or dropped on each network interface since the last device reset.

The following are some of the ENA metrics:

  • bw_in_allowance_exceeded: The number of packets queued or dropped because the inbound aggregate bandwidth exceeded the maximum for the instance.
  • bw_out_allowance_exceeded: The number of packets queued or dropped because the outbound aggregate bandwidth exceeded the maximum for the instance.
  • pps_allowance_exceeded: The number of packets queued or dropped because the bidirectional Packets Per Second (PPS) exceeded the maximum for the instance.
  • conntrack_allowance_exceeded: The number of packets queued or dropped because the network traffic for the instance exceeded the maximum number of connections that can be tracked.

Network performance specifications vary according to the instance type. For more information on network performance specifications, see Amazon EC2 instance types. In some cases, you might see queuing or drops even though your average bandwidth or PPS as seen in Amazon CloudWatch is low. For example, the NetworkIn, NetworkOut, NetworkPacketsIn, or NetworkPacketsOut metrics in CloudWatch show amounts that don't suggest a limit being reached.

Resolution

Microbursts are the most common cause of the preceding symptoms. Microbursts are short spikes in demand followed by periods of low or no activity. Microbursts typically last only for seconds, milliseconds, or even microseconds. The CloudWatch metrics listed in the previous section aren't granular enough to reflect microbursts.

Average calculation

The EC2 metrics in CloudWatch listed in the previous section are sampled every 60 seconds. These metrics capture the total bytes or packets transferred in that period. These samples are then aggregated and published to CloudWatch in 5-minute periods. Each statistic in the period returns a different sample value:

  • Sum is the sum of all sample values.
  • SampleCount is the number of aggregated samples. In the previous example, there are 5 samples.
  • Minimum is the sample value with the lowest byte or packet count.
  • Average is the average sample value, calculated by dividing Sum by SampleCount.
  • Maximum is the sample value with the highest byte or packet count.

You can calculate the average throughput or PPS in two ways:

  • Divide Sum by Period. For example, divide the sum by 300 seconds for a simple 5-minute average.
  • Divide Maximum by 60 seconds for an average in the minute with the highest activity.

Microbursts in CloudWatch metrics

The following is an example of a microburst, and how it appears in CloudWatch:

  • The instance has a network bandwidth performance of 10 Gbps (1.25 GB/s).
  • In a given sample of 60 seconds, an outbound data transfer of 20 GB uses up all available bandwidth. This action increments bw_out_allowance_exceeded. Transfer completes in about 20 seconds, and no further data is sent.
  • Instance remains idle for the remaining 4 samples of 240 seconds.

In this example, the average throughput in a 5-minute period is much lower than the one during the microburst:

SUM(NetworkOut) / PERIOD = ((20 GB \* 1 sample) + (0 GB \* 4 samples)) / 300 seconds = ~0.066 GB/s \* 8 bits = ~0.533 Gbps

Even if you calculate the throughput based on the highest sample, the average still doesn't reflect the throughput amount:

MAXIMUM(NetworkOut) / 60 = 20 GB / 60 seconds = ~0.333 GB/s \* 8 bits = ~2.667 Gbps

Monitoring microbursts

To measure throughput and PPS at a more granular level, use operating system (OS) tools to monitor network statistics. Monitor your network statistics more frequently during periods of shaping or high activity.

The following are examples of OS tools that you can use:

Linux

  • sar
  • nload
  • iftop
  • iptraf-ng

Windows

  • Performance Monitor

The CloudWatch agent can be used on both Linux and Windows to publish OS-level network metrics to CloudWatch as custom metrics. These metrics can be published at intervals as low as 1 second. High-resolution metrics with a period less than 60 seconds lead to higher charges. For more information about CloudWatch pricing, see CloudWatch pricing.

It's a best practice to monitor the network performance metrics provided by ENA. To view the metrics, the driver version must be greater than or equal to 2.2.10 (Linux) and 2.2.2 (Windows. For more information, see Monitor network performance for your EC2 instance.

CloudWatch Agent can also publish ENA metrics. For instructions on publishing ENA metrics for Linux, see Collect network performance metrics. For Windows, ENA metrics are available in Performance Monitor. You can configure the CloudWatch Agent to publish available metrics in Performance Monitor.

Preventing microbursts

To avoid microbursts, traffic must be paced at the senders so that the traffic doesn't exceed a maximum throughput or packet rate. This makes microbursts difficult to avoid. To pace traffic at the senders, application-level changes are required. Depending on how this change is implemented, OS support for traffic pacing might need to be available and then turned on at the senders. However, that configuration might not always be possible or practical.

Microbursting can also happen because too many connections send packets in a short period. When this happens, pacing individual senders doesn't prevent this issue.

It's a best practice to monitor ENA metrics. If limits are reached often, or if there's evidence that your application is affected by traffic shaping, then do the following:

  • Scale up: Move to a larger instance size. Larger instances have higher allowances. Network optimized instances, such as those with an "n" like C5n, have higher allowances than their non-network-optimized counterparts.
  • Scale out: Spread traffic across multiple instances to reduce traffic and contention at individual instances.

For Linux-based operating systems, there are additional mitigation options for advanced users. You can implement the following options alone or in combination. It's a best practice to benchmark mitigations in a testing environment to verify that they reduce or eliminate traffic shaping without adverse effects on the workload.

SO_MAX_PACING_RATE

This socket option can be passed by an application to the setsockopt system call to specify a maximum pacing rate (bytes per second). The Linux kernel then paces traffic from that socket so that the traffic doesn't exceed the limit. This option requires the following:

  • Application code-level changes.
  • Support from the kernel. For more information, see net: introduce SO_MAX_PACING_RATE on the GitHub website.
  • The use of Fair Queue (FQ) queuing discipline or the kernel's support for pacing at the TCP layer (applicable to TCP only).

For more information, see getsockopt(2) - Linux manual page and tc-fq(8) - Linux manual page on man7.org and tcp: internal implementation for pacing on the GitHub website.

Queuing disciplines (qdiscs)

qdiscs are responsible for packet scheduling and optional shaping. Some qdiscs, such as fq, help smooth out traffic bursts from individual flows. For more information, see the tc(8) - Linux manual page manual page on man7.org.

Shallow Transmission (Tx) queues

In some scenarios, shallow Tx queues help reduce PPS shaping. This reduction can be achieved in two ways:

  • Byte Queue Limits (BQL): BQL dynamically limits the number of in-flight bytes on Tx queues. BQL is turned on by default on ENA driver versions shipped with the Linux kernel that ends with a "K". For ENA driver versions from GitHub that end with a "g", BQL is available in versions v2.6.1g and later, and is turned off by default. You can turn on BQL using the enable_bql ENA module parameter. For more information, see bql: Byte Queue Limits on lwn.net.
  • Reduced Tx queue length: Reduce Tx queue length from its default of 1,024 packets to a lower amount (minimum 256). You can change this length using the ethtool. For more information, see ethtool(8) - Linux manual page on man7.org.

Related information

Amazon EC2 instance network bandwidth

AWS OFFICIAL
AWS OFFICIALUpdated 7 months ago
7 Comments

will the packets that will get dropped or queued due to the allowance pps_allowance_exceeded exceeding also be counted into the metric "NetworkPacketsIn"? Therefore I could see how many packets per second (PPS) were coming in for the allowance to exceed. This information from the production environment could be used to simulate load using iperf.

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
EXPERT
replied a year ago

will the packets that will get dropped or queued due to the allowance pps_allowance_exceeded exceeding also be counted into the metric "NetworkPacketsIn"?

Dropped packets do not count towards CloudWatch's Network* metrics. Queued packets do count, though, because they eventually go through.

profile pictureAWS
replied a year ago

Is it possible to differentiate whether packets are queued or dropped? Dropped seems like a more serious condition.

profile picture
EXPERT
replied 7 months ago

Is it possible to differentiate whether packets are queued or dropped?

For TCP connections, queuing results in higher Round Trip Time (RTT), and drops result in retransmissions. To monitor these, you must do it at OS level. The nstat tool displays many useful system-level SNMP counters for TCP. And with ss, you can obtain similar statistics at a socket level. On the fancier side, there are tools out there which leverage perf or eBPF for similar purposes. These days, I prefer tcpretrans and tcprtt from the BPF Compiler Collection (BCC).

Dropped seems like a more serious condition.

In a sense, yes. However, packets may be dropped at many layers within and beyond EC2 instances for many reasons, meaning that packet drops are normal and expected to an extent. Therefore, metrics like these shouldn't be used by themselves when measuring business impact. You must always correlate them with one or more Key Performance Indicators (KPIs) from your application to determine their true impact.

profile pictureAWS
replied 3 months ago

is there any value in watching the interface metrics for dropped packets at the "ifconfig" level like below:

    RX packets 395480  bytes 149047230 (142.1 MiB)
    RX errors 0  dropped 0  overruns 0  frame 0
    TX packets 274803  bytes 47960383 (45.7 MiB)
    TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Would that be a good indicator if packets are being dropped due to bandwidth allowances ( or other factors )

replied 3 months ago

is there any value in watching the interface metrics for dropped packets at the "ifconfig" level like below: Would that be a good indicator if packets are being dropped due to bandwidth allowances ( or other factors )

There may be value in monitoring those, but not because of bandwidth allowances. These counters reflect packet drops happening at OS level for various reasons. On Receive (Rx) side, packets dropped due to exceeding EC2 instance network allowances don't make it to the instance, therefore they aren't reflected as drops in ethtool or ip (note that ifconfig is obsolete). Likewise, on Transmit (Tx) side, packets dropped by the OS don't make it to EC2. Information about these counters is available at Interface statistics and other sources. For Rx drops in particular, Red Hat has 3 articles covering this topic in detail (subscription required):

profile pictureAWS
replied 3 months ago