Inter-AZ Latency for SAP

Overview

When it comes to deploying SAP solutions on AWS (RISE or Native AWS), one of the critical aspects to consider is the latency between Availability Zones (AZs). Customers often inquire about the existing latency between AZs to determine whether it's suitable for a distributed SAP installation, which can help with resilience, failover, and high availability/disaster recovery (HA/DR). This also helps in providing higher performance for completion of mission critical jobs by relocating or running them on AZ Pair having lesser network latency.

Background

In the networking section of the "Architecture Guidance for Availability and Reliability of SAP on AWS" documentation, it is highlighted that having application servers in a different AZ than the database may impact performance. Suggestions are provided, such as the use of load balancing, to locate latency-sensitive users or batch processes in the same AZ as the database.

Purpose

SAP provides guidance on acceptable network latency for the performance of their products. For example, SAP Note 1100926 (FAQ: Network performance) and SAP Note 2879613 (ABAPMETER in NetWeaver AS ABAP) discuss latency thresholds between SAP application servers and the database. Generally, for performance-critical LAN environments (e.g., a server network between SAP and the database), you can assume that the following threshold values apply: Good value: Roundtrip time <= 0.3 ms Moderate value: 0.3 ms < roundtrip time <= 0.7 ms Below average value: Roundtrip time > 0.7 ms

For WAN environments, considerably longer roundtrip times are often acceptable.

Please look at SAP Note - 3496343 - Network Latency on AWS.

Why Measure Inter-AZ Latency?

Measuring inter-AZ latency is crucial for several reasons:

1. Performance: Latency can significantly impact the performance of SAP applications, especially for latency-sensitive workloads. By understanding the latency between AZs, customers can make informed decisions about the placement of their SAP components to optimize performance.
2. Resilience and HA/DR: Distributing SAP components across multiple AZs can enhance resilience, failover capabilities, and HA/DR strategies. However, excessive latency between AZs can undermine these benefits, making it essential to evaluate inter-AZ latency before implementing such architectures.
3. Capacity Planning: Latency measurements can inform capacity planning decisions, such as sizing compute resources, network configurations, and workload distribution across AZs.

Solution

To measure inter-AZ latency, customers can leverage the niping tool recommended by SAP in Note 2879613. This tool provides a convenient way to gauge the latency between SAP components, including application servers and databases. Additionally, using AWS Network Manager ,customers can now monitor the network latency between AWS regions, Availability Zones and within each availability zones. AWS Network Manager offers an easier and quicker way to assess latency without the need for manual measurements.Please note Infrastructure Performance does not incorporate performance metrics for paths through VPC networking resources, such as transit gateways, NAT gateways, VPC endpoints, Elastic Load Balancing, or Amazon EC2 network interfaces. Please see detailed document

Avoiding Network Latency for Large EC2 Instance Types

When using large EC2 instance types, such as those often employed for SAP HANA but can be leveraged for large batch/application instances, network latency can be significantly affected by processor c-states (CPU power states/sleep settings). Instances with c-state control may require specific configurations to minimize latency. AWS provides a list of instance types that have c-state control, such as the compute-optimized instances:

c5.9xlarge | c5.12xlarge | c5.18xlarge | c5.24xlarge | c5a.24xlarge | c5ad.24xlarge | c5d.9xlarge | c5d.12xlarge | c5d.18xlarge | c5d.24xlarge | c5n.9xlarge | c5n.18xlarge | c6a.24xlarge | c6a.32xlarge | c6a.48xlarge | c6i.16xlarge | c6i.32xlarge | c7i.large | c7i.xlarge | c7i.2xlarge | c7i.4xlarge | c7i.8xlarge | c7i.12xlarge | c7i.16xlarge | c7i.24xlarge | c7i.48xlarge

For these instances, adjusting the c-state settings can help reduce network latency.

Customer example demonstrated how adjusting the c-state settings in Red Hat Enterprise Linux 8.4 improved network latency. The Red Hat log showed:

[    9.434985] intel_powerclamp: No package C-state available

[    9.522647] intel_powerclamp: No package C-state available

2023/09/27 22:42:43Z: Amazon SSM Agent v3.2.985.0 is running

2023/09/27 22:42:43Z: OsProductName: Red Hat Enterprise Linux

2023/09/27 22:42:43Z: OsVersion: 8.4

[    9.594708] intel_powerclamp: No package C-state available

[    9.785520] intel_powerclamp: No package C-state available

[    9.961496] intel_powerclamp: No package C-state available

Before c-state configuration:

traceroute to 10.97.86.145 (10.97.86.145), 30 hops max, 60 byte packets

1  10.97.86.145  **0.720 ms**  0.749 ms  0.729 ms

After adding the following lines to the GRUB_CMDLINE_LINUX in /etc/default/grub ((take a backup of the file) and add the values don't remove anything thats there)

processor.max_cstate=1 intel_idle.max_cstate=1

And then:

grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the system.

The customer observed significant reduction in network latency :

traceroute to 10.97.86.145 (10.97.86.145), 30 hops max, 60 byte packets

1  10.97.86.145  **0.133 ms**  0.124 ms  0.110 ms

IMPORTANT NOTE:

This solution is applicable for RISE Deployments as well .

Conclusion

Measuring and understanding inter-AZ latency is crucial for deploying SAP solutions on AWS, particularly in distributed architectures or when implementing resilience, failover, and HA/DR strategies. By leveraging tools like niping and the AWS Network Manager, customers can assess latency and make informed decisions about component placement, capacity planning, and network configurations. Additionally, for large EC2 instance types, customers should be mindful of the potential impact of processor c-states on network latency and consider adjusting these settings as necessary to optimize performance. By carefully evaluating inter-AZ latency and implementing appropriate measures, customers can ensure optimal performance, resilience, and reliability for their SAP deployments on AWS.

Additional Readings:

1. SAP Lens
2. AWS for SAP Blog - Automate and Optimize SAP Network Performance in a Multi-AZ deployment

Co-author : Ramkrishna Borhade