How can I find the network PPS limit for Amazon EC2 instances?

3 minute read
0

I have an Amazon Elastic Compute Cloud (Amazon EC2) instance, and I want to find the network packets per second (PPS) limit for that instance.

Short description

The PPS for an EC2 instance depends on several network characteristics for the instance. These characteristics include the following factors:

  • Traffic mix, TCP versus UDP
  • Number of flows
  • Packet size
  • New connections versus existing connections
  • Applied security group rules

PPS allowance is separately considered from the overall bandwidth allowance. Though an instance might be under overall bandwidth allowance, you can exceed the PPS allowance if the mean packet size is too small. To determine if your instance currently exceeds the PPS allocated, extract the information from the ENA driver.

Resolution

To perform a PPS benchmark, use an iperf2 test. Depending on the network characteristics, customers can use the iPerf benchmark to determine the approximate PPS limit of any EC2 instance.

To perform the benchmark, complete the following steps:

  1. Use an Amazon Linux 2 or Amazon 2023 image to create two EC2 instances in the same Availability Zone.
    Note: The instances must be the same instance type.

  2. To install development tools and git on both instances, run the following commands:

    sudo yum groupinstall "Development Tools" -y
    sudo yum install git -y
  3. Run the following commands to change the directory to the /usr/local directory and clone iPerf code from Git to both EC2 instances:

    cd /usr/local/
    sudo git clone https://git.code.sf.net/p/iperf2/code iperf2-code
  4. To build and install the package on both EC2 instances, run the following commands:

    cd /usr/local/iperf2-codesudo 
    ./configure
    sudo make && make install
  5. To have the first instance act as the iPerf server running in UDP mode for maximum performance, run iPerf:

    sudo /usr/local/bin/iperf -s -u

    The command returns the following output:

    ------------------------------------------------------------
    Server listening on UDP port 5001
    UDP buffer size:  208 KByte (default)
    ------------------------------------------------------------
  6. For the iPerf client to connect successfully to the iPerf server, make sure that the security group associated to the server allows connections on 5001/UDP. For more information, see Security group example.

  7. On the second instance, use the iPerf client to connect with UDP to the iPerf server:

    sudo /usr/local/bin/iperf -c <private_ip_of_server_instance> -u -i 1 -l 16 -b 20kpps -e -P64 -o /tmp/bw_test.txt && grep -i sum-64 /tmp/bw_test.txt

    Note: Replace <private_ip_of_server_instance> with the internal IP address of the iPerf server.

The output of the command is written to a log file under /tmp/bw_test.txt. The summary of each second the test is run is displayed, as well as the PPS.

The command returns the following output:

[SUM-64] 0.00-1.00 sec 9.42 MBytes 79.1 Mbits/sec 617416/2 617416 pps
[SUM-64] 1.00-2.00 sec 10.1 MBytes 84.3 Mbits/sec 658689/0 658692 pps
[SUM-64] 2.00-3.00 sec 10.1 MBytes 84.7 Mbits/sec 661837/0 661838 pps
[SUM-64] 3.00-4.00 sec 10.1 MBytes 84.6 Mbits/sec 661226/0 661226 pps
[SUM-64] 6.00-7.00 sec 9.73 MBytes 81.7 Mbits/sec 637975/0 637975 pps
[SUM-64] 7.00-8.00 sec 9.46 MBytes 79.4 Mbits/sec 620172/0 620172 pps
[SUM-64] 8.00-9.00 sec 9.46 MBytes 79.4 Mbits/sec 620150/0 620151 pps

This output shows the maximum PPS achieved for this instance type. For more information, see IPERF on the iPerf website.

Related information

iPerf2 user documentation on the iPerf website

AWS OFFICIAL
AWS OFFICIALUpdated 17 days ago
5 Comments

Can you please provide an example of an output with its interpretation?

replied 9 months ago

I am seeing the following output for e.g: Stripped off the initial part which shows the ip address.

[ ID] Interval Transfer Bandwidth Write/Err PPS [ 12] 0.00-1.00 sec 83.1 KBytes 681 Kbits/sec 0/0 5316 pps [ 48] 0.00-1.00 sec 80.8 KBytes 662 Kbits/sec 0/0 6313 pps [ 16] 0.00-1.00 sec 78.1 KBytes 640 Kbits/sec 0/0 4995 pps ....

[SUM] 0.00-10.03 sec 57.9 MBytes 48.4 Mbits/sec 0/0 378161 pps [SUM-128] Sent 3791670 datagrams [ 8] Server Report: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 8] 0.00-9.76 sec 837 KBytes 703 Kbits/sec 0.020 ms 6786/60369 (11%) [ 8] 0.00-9.76 sec 2 datagrams received out-of-order [ 6] Server Report: [ 6] 0.00-10.02 sec 818 KBytes 669 Kbits/sec 0.018 ms 7630/59966 (13%) [ 6] 0.00-10.02 sec 165 datagrams received out-of-order .... [ 9] Server Report: [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 9] 0.00-10.01 sec 848 KBytes 694 Kbits/sec 0.791 ms 4281/58556 (7.3%)

I see the total datagrams sent and the loss per thread. Where do I see the pps limit?

replied 9 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 9 months ago

Also if we test this on a windows instance, with iperf3, although there is a packet loss observed in the iperf output, we don't see any allowance_exeeded metrics in Windows Performance Monitor - ENA packet reshaping reflecting this. https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/monitoring-network-performance-ena.html

replied 8 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 8 months ago