How can I review NUMA statistics for my workload running on EC2 instances?

Short description

There are multiple factors that can affect the performance of the application hosted on EC2 instance including CPU over utilization, memory use, the EBS volume, network statistics, or if the application isn't NUMA aware. NUMA is non-uniform memory access. In NUMA architecture, each CPU has access to its own assigned memory, known as local memory. Each CPU can also access memory allocated to other CPUs, known as foreign memory. If applications hosted on your instances aren't NUMA aware, then accessing the foreign memory incurs some additional costs and might affect performance.

Contact the application vendor to confirm if your application is NUMA aware.

Resolution

Review NUMA statistics

Note: To review the components of performance other than NUMA statistics, see How do I troubleshoot slow connections to a website hosted on my EC2 instance?

To review NUMA statistics, do the following:

1.    Run the following command to confirm if the instance type is in single NUMA or multi-NUMA node.

Note: The instance type is r5.16xlarge has 2 NUMA nodes.

lscpu | grep -i numa
NUMA node(s): 2

2.    Run the following command to install the numactl package:

sudo yum install numactl

3.    Run the following command to review the NUMA topology:

sudo numactl -H

In the following output example the NUMA topology is divided into two nodes, node 0 and node 1. Node 0 has 32 CPUs and 255,225 MB of memory assigned to it. Node 1 has the other half of the CPUs (32) and 254,924 MB of assigned memory. The node distance indicates the latencies involved in accessing memory pages from the other node.

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 255140 MB
node 0 free: 254794 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
node 1 size: 255225 MB
node 1 free: 254924 MB
node distances:
node   0   1
0:  10  21
1:  21  10

4.    Run the following command to review the NUMA statistics:

sudo numastat

The following is an example of the command output:

node                       node0           node1
numa_hit                  314825          288025
numa_miss                      0               0
numa_foreign                   0               0
interleave_hit             37958           37620
local_node                311752          248476
other_node                  3073           39549

For a detailed explanation of each statistic, see A.11. NUMASTAT on the Red Hat Customer Portal.

Increases in the numa_foreign and numa_miss statistics might indicate that the application running on the EC2 instance isn't NUMA aware. This might affect performance. To resolve this issue, disable NUMA at the grub level. Or, try binding the application to the specific NUMA node. For more information, see step 8 in Operating system optimizations.

Disable NUMA permanently

To disable numa permanently on Amazon Linux 2, CentOS7, or RHEL 7, run the following commands:

sudo vi /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="numa=off
grub2-mkconfig -o /etc/grub2.cfg
reboot