Attaching Amazon FSx for NetApp ONTAP NFS Datastore(s) as secondary storage for an Amazon Elastic VMware Service environment.

Background

Amazon Elastic VMware Service (Amazon EVS), an AWS service that enables you to run VMware Cloud Foundation (VCF) within your Amazon Virtual Private Cloud (VPC), now integrates with Amazon FSx for NetApp ONTAP (FSxN), a fully managed AWS service built on NetApp’s ONTAP file system. With this integration, Amazon EVS customers can use FSxN as a scalable, high-performance external datastore for their VCF environments on AWS, enabling them to benefit from cost savings through independent scaling of storage and compute, and automated data tiering. If you are running VMware with ONTAP on-premises, you can use the same tools and workflows to easily migrate those workloads to AWS and maintain operational consistency. The Amazon EVS integration with FSxN is now available for public preview in all AWS Regions where both Amazon EVS and FSxN are available.

This article will walk you through attaching Amazon FSxN NFS volumes as secondary storage for your Amazon EVS deployment and how to apply NetApp best practice settings where appropriate. It is recommended to read the entire article and design considerations before attaching FSxN NFS datastores to your Amazon EVS environment.

Situation

If you are in a situation where you become storage bound within a cluster running within your Amazon EVS environment, you can increase your storage footprint without having to add new ESXi hosts. You can also deploy FSxN datastores with certain defined performance characteristics e.g. with different levels of IOPS or throughput, for storage tiering within your Amazon EVS environment.

Note: As NetApp ONTAP provides multi-protocol storage access, you can opt to add volumes from your FSxN file system and mount to your ESXi hosts via iSCSI. The scope of this article is NFS only, and iSCSI will be covered in a separate article.

Tasks

1. Create an FSxN file system and storage volumes from the AWS console or CLI
2. Enable VMware vStorage for NFS on the Storage Virtual Machine (SVM)
3. Ensure that vStorage APIs for Array Integration (VAAI) is enabled on each ESXi host
4. Download, install & configure the NetApp NFS Plug-in for VMware VAAI
5. Apply NetApp ONTAP recommended ESXi host settings for NFS storage
6. Mount your FSxN volume in your Amazon EVS environment
7. Performance tuning – nConnect and jumbo frames

Actions

1. Create an FSxN file system and storage volumes from the AWS console or CLI

Step 1: Using the AWS console or CLI, navigate to Amazon FSx and choose Create a file system. Next, select Amazon FSx for NetApp ONTAP from within the file system options.

Use the standard create method and define the File system details, Network and security, Encryption, Default storage virtual machine configuration, Default volume configuration, Default volume storage tiering and Default volume SnapLock configuration as per your use case requirements.

Find detailed guidance for getting started with Amazon FSx for NetApp ONTAP here.

To make life easier in the upcoming steps, it is highly recommended that you set a password for the File system and Storage virtual machine where prompted during creation, as it can be time consuming to do this post deployment.

Step 2: Once the FSxN file system is deployed, create any volumes you wish to use as ESXi host storage as per your requirements. If you do not want to use the default SVM to serve data to your Amazon EVS environment, create a new one per your requirements. Find instructions here.

Note: At GA, Amazon EVS will support a single-AZ environment deployment only. You can deploy your FSxN file system in multi-AZ configuration if desirable, but in the event of an AZ failure where your Amazon EVS environment is running, you will need to deploy a brand new EVS environment in the other AZ where your secondary file system is running; manually mount the NFS volumes to each ESXi host in that environment and import all VMs into the vCenter inventory via their VMX file.

There will be cost implications to running in FSxN in a multi-AZ setup. For more information on Amazon FSx for NetApp ONTAP pricing, please read here.

Design Consideration: You can deploy your FSxN file system within your Amazon EVS service access subnet if desired, or another standard VPC subnet within your Amazon EVS VPC or any VPC subnet connected over a Transit Gateway (TGW) VPC attachment (Consider latency and number of hops with TGW approach).

You cannot use one of the Amazon EVS VLAN subnets created during Amazon EVS environment deployment.

If a new subnet is created for FSxN usage, ensure it is added into the VPC route table used by Amazon EVS and appropriate security is applied to lock down access.

2. Enable VMware vStorage for NFS on the Storage Virtual Machine (SVM)

By default, VMware storage support over NFS is disabled on SVMs. Therefore, you will need to enable vStorage support for NFS manually on any SVMs being used to present NFS storage to your ESXi hosts.

Step 1: From the Amazon FSxN console, on the left under ONTAP select Storage virtual machines, then select the SVM name of the SVM you will be using to provision storage into your Amazon EVS environment.

Figure 1. Storage virtual machine view.

Step 2: Next, select Administration and note the Management IP address and SVM administrator username – You will use these in the next step.

Figure 2. Storage virtual machine Administration view.

Step 3: SSH into the SVM using the IP address and SVM vsadmin username from the previous step. The password was set during the creation of the FSxN file system.

Step 4: Run the following command on the SVM to enable VMware Storage over NFS:

vserver nfs modify -vstorage enabled

Step 5: Run the following command on the SVM to validate that VMware Storage over NFS is now enabled:

vserver nfs show -instance

Figure 3. ESXi SSH view with sample output validating NFS vStorage Support is enabled.

3. Ensure that vStorage APIs for Array Integration (VAAI) is enabled on each ESXi host:

VAAI is a set of VMware APIs that allow ESXi hosts to offload certain storage operations to compatible storage arrays. This offloading improves performance by reducing the load on the ESXi host CPU and the storage network, leading to faster operations for VM cloning, provisioning, and storage vMotion.

Note: You may need to temporarily enable SSH access on each ESXi host from within the vCenter web client to accomplish this and other steps listed out in this article. Instructions are available here.

Step 1: SSH into each host that you will connect to the FSxN file system and run the following commands to check that VAAI is enabled. This should be enabled by default, but is an important validation step.

esxcfg-advcfg -g /DataMover/HardwareAcceleratedMove

esxcfg-advcfg -g /DataMover/HardwareAcceleratedInit

Both commands should return a value of 1. If they do, you can skip Step 2.

Step 2: If they return a value of 0, run the following commands to enable VAAI:

esxcfg-advcfg -s 1 /DataMover/HardwareAcceleratedInit

esxcfg-advcfg -s 1 /DataMover/HardwareAcceleratedMove

Figure 4. ESXi SSH view with sample output validating that VAAI has been enabled

4. Download, install & configure the NetApp NFS Plug-in for VMware VAAI

The NetApp NFS VAAI plugin is a software component that enables VMware ESXi hosts to offload certain storage operations to a NetApp NFS storage array.

Step 1: Download the NetApp NFS Plug-in for VMware VAAI from the NetApp support website here.

Step 2: Transfer the plugin to each host within your Amazon EVS environment. You can use an SCP tool of your choice or the CLI of your workstation to accomplish this. An example command from a Windows command prompt is below:

C:\Users\Administrator\Downloads>scp NetAppPlugin2.0.1.zip root@esxi01.domain.name:/tmp

Figure 5. Example of a successful upload of the NetApp NFS plug-in using SCP.

Step 3: Once the plugin has been uploaded, you will need to stop the vaai-nasd service on the host, install the plugin & then start the vaai-nasd service back up. Use the following commands on each host to do this:

/etc/init.d/vaai-nasd stop

esxcli software component apply -d /tmp/NetAppNasPlugin2.0.1.zip

/etc/init.d/vaai-nasd start

Step 4: Validate that the plugin is successfully installed on each host by running the following command:

esxcli software component list | grep NetApp

Figure 6. ESXi SSH view validating the NetApp NFS plug-in has been successfully installed.

5. Apply NetApp ONTAP recommended ESXi host settings for NFS storage

NetApp provides various recommended best practice settings to apply to ESXi hosts for optimizing performance, depending on storage protocols used within your vSphere environment e.g. NFS, iSCSI, FC.

Since this article covers attaching the datastores from FSxN via NFS, we recommend applying the NetApp NFS settings to each ESXi host within your Amazon EVS environment.

There are 2 approaches to accomplish this:

Approach 1: You can download and deploy the NetApp provided ONTAP tools for VMware vSphere OVA into your Amazon EVS environment and configure for your vCenter. Once deployed, there are options to apply best practices automatically to each ESXi host within your environment. Find instructions here.

Approach 2: You can apply these recommended settings to each ESXi host in your Amazon EVS environment manually. Follow through the NetApp documentation here and change the listed ESXi Advanced Configuration and NFS Settings to the NetApp recommended values.

Important: You may be required to reboot each host after applying these settings to take effect. Reboot time can take up to 30 minutes, so you should ensure that the host you are working on is back online and out of maintenance mode within vCenter before working on the next host. This will help avoid potential issues with vSAN storage availability within your cluster if multiple ESXi hosts are down at the same time.

6. Mount your FSxN volume in your Amazon EVS environment

Step 1: From the FSxN console, on the left under ONTAP select Storage virtual Machines, then select the SVM name of the SVM you will be using to provision your Amazon EVS NFS storage.

Step 2: Select the Endpoints tab and note the NFS DNS name and NFS IP address, then choose the Volumes tab and note the Volume name.

Figure 7. Storage virtual machine Endpoints view,

Figure 8. Storage virtual machine Volumes view

Now that you have your FSxN volume and endpoint connectivity details, you will need to switch over to your Amazon EVS environment to mount the volumes as NFS datastores on your ESXi hosts.

Step 3: Log in to your vCenter web client and enter the Host and Clusters view.

Step 4: Enter the context menu of the cluster of hosts you want to mount the FSxN volumes to, then select Storage then New Datastore.

Step 5: Select NFS.

Step 6: Select the NFS version you wish to use when mounting the volume on the ESXi Hosts. In our example, we are using NFS v4.1.

Note: ESXi 8.0 Update 3 onward has support for both NFS v3 and v4.1. Both of these NFS versions are supported by FSxN, so the choice of which version to use will come down to your requirements.

For more detail on NFS versions and their differences, find Broadcom documentation on best practices running NFS in VMware vSphere here.

TL;DR NFS v3 is generally considered well supported and highly compatible with all ESXi storage features, but traffic is sent in clear text and volumes are mounted with root privileges. NFS v4.1 offers security authentication and encryption with Kerberos, and advanced multi-pathing options, but is not compatible with things like storage I/O control or storage DRS.

Step 7: Provide a Datastore name, and enter the Folder and the Server information. These values are the Volume name for Folder and either NFS DNS name or NFS IP address for Server recorded in Step 2.

Step 8: Select Don’t use Kerberos authentication. In this example we are not using Kerberos. If you are using it to connect to your NFS volumes, you will need to select Kerberos authentication and specify relevant credentials.

Step 9: Select the hosts you wish to connect to the datastore.

Step 10: Review and then select Finish.

Figure 9. vCenter view with example datastore added and ready to be mounted.

Design Consideration: By default, the ESXi hosts deployed within an Amazon EVS environment will connect to and mount the NFS volumes over the management VLAN VMkernel interface (vmk0).

For production environments it would be highly recommended to utilize one of the expansion VLAN subnets and create a new distributed port group and VMkernel interface for purposes of segregating NFS storage traffic. This helps not only to provide an extra layer of security by ensuring NFS storage traffic is isolated to its own VLAN, but also mitigates potentially overloading the management VMkernel interface with high I/O, which in turn could cause vSphere HA related false positives or other related management plane issues. This approach can also aid with general troubleshooting of traffic flow by providing a logical separation of different traffic types across dedicated interfaces within your Amazon EVS environment. An article will be published on how to accomplish this shortly.

Once a new VMkernel port has been created for storage purposes, you can then bind NFS datastores to use that interface when mounting the volumes. Find Broadcom instructions for VMkernel Binding for NFS v3 or v4.1 Datastores here.

In lieu of using separate VMkernel ports, for additional security when connecting to NFS storage over vmk0, you can enable and use Kerberos to provide secure authentication when connecting to your NFS datastores. You will need to ensure you are connecting to your volumes using NFS v4.1, as this authentication type is not supported in NFS v3. Find NetApp documentation on Kerberos with NFS on ONTAP here and Broadcom using Kerberos for NFS 4.1 with ESXi here.

Tip: Ensure that the Security Group attached to the FSxN file system allows access from the IP addresses set on the ESXi host interfaces you will be connecting from.

7. Performance tuning

nConnect

nConnect is a feature that allows for multiple TCP connections to be established between an ESXi host and an NFS server; for a single NFS datastore. This enables increased parallelism and better performance by distributing I/O operations across these connections. Essentially, it's a way to improve the throughput and efficiency of NFS based storage when used with VMware vSphere.

By default, there is 1 TCP connection per volume used to transfer data between the host and the FSxN datastore.

Step 1: To increase the number of TCP connections, connect to an ESXi host within your Amazon EVS environment using SSH and enter the following nConnect command, replacing fsxn with the name of your FSxN datastore as it appears in your ESXi or vCenter storage view from the vCenter web client:

NFS v3: esxcli storage nfs param set -v fsxn -c 4

Or

NFS v4.1: esxcli storage nfs41 param set -v fsxn -c 4

This command will enable 4 concurrent connections to the datastore from the ESXi host.

If you wish to increase this to a maximum value of 8, you can run an advanced command found in the Broadcom KB here before running the parameter set command:

Step 1: esxcfg-advcfg -s 8 /NFS/MaxConnectionsPerDatastore

Step 2: NFS v3: esxcli storage nfs param set -v fsxn -c 8

Or

NFS v4.1: esxcli storage nfs41 param set -v fsxn -c 8

Repeat this step for each ESXi host within your Amazon EVS environment which has the volume mounted as an NFS datastore.

Design Consideration: As the number of connections per NFS datastore increases, the number of NFS datastores which can be mounted on that ESXi host decreases. If you are planning on adding multiple NFS datastores, this should be considered when defining the maximum number of connections per ESXi host for each volume, to ensure that you do not hit defined limits.

As of writing, the total maximum number of connections supported per ESXi host is 256, so if you were to use 8 connections per volume, you will be limited to 32 NFS datastores mounted to each ESXi host.

Check Broadcom KB article 91481 for additional details.

Modify Maximum Transmission Unit (MTU) to 8500 bytes for jumbo frames

Design Consideration: MTU refers to the largest size of a data packet (in bytes) that can be transmitted across a network without fragmentation. A common MTU size is 1500 bytes, often used in Ethernet networks. Jumbo frames are Ethernet frames with payloads larger than the common 1500 bytes. For storage networks, the increase in frame size allows for more data to be transmitted in a single frame, reducing the overhead associated with processing numerous smaller frames and potentially improving overall storage network performance. Any MTU defined on the ESXi host vSwitches and VMkernel ports must have the same value throughout the entire networking stack to avoid packet fragmentation.

To get the benefits of jumbo frames from your Amazon EVS environment and its network connection to FSxN based storage, you will need to update the MTU value in the following locations:

ESXi host vmk0

If you are using the management VLAN vmk0 to mount your NFS datastores, by default, the MTU is set to 1500 bytes. This can be updated to a value of 8500 bytes, which is the largest frame size currently supported end to end within the Amazon EVS environment.

To do this, perform the following actions:

Step 1: From your vCenter web client, Hosts and Clusters view, select an ESXi host in your inventory and choose the Configure tab from the right-hand plane.

Step 2: Under Networking, select VMkernel adapters, and select the 3 dots context menu next to vmk0 and select Edit.

Step 3: From the port properties section, update the MTU (Bytes) value from 1500 to 8500. Select OK.

Step 4: Perform the same steps for the rest of the ESXi hosts in your Amazon EVS environment.

Figure 10. vCenter view displaying example of editing vmk0 settings

Figure 11. vCenter view with example of updating MTU from 1500 to 8500 bytes

Note: If you are binding a separate/dedicated VMkernel port utilizing one of your expansion VLAN subnets when mounting your datastore, ensure that during creation the MTU of the VMkernel port is set to 8500 bytes. By default, this should inherit the vSwitch MTU value which is already set to 8500 bytes.

FSxN broadcast domain

FSxN file system broadcast domains contain groups of logical network ports bound within the FSxN storage cluster. These ports are used by SVMs for various network functions like serving data. The MTU for both the Default and Fsx broadcast domains will be set to 9001 bytes by default.

Step 1: You can confirm this by connecting to your SVM over SSH using the fsxadmin user, and running the following command:

network port show

You will see the MTU listed out per port.

It is recommended to change the MTU for the Fsx broadcast domain which serves storage traffic to your ESXi hosts to 8500, to match what is set throughout the Amazon EVS environment.

Step 2: Enter the following command to update, substituting Fsx with your file system name if required:

broadcast-domain modify -broadcast-domain Fsx -mtu 8500

Figure 12. SVM CLI view with example output updating MTU to 8500 bytes on the Fsx broadcast domain

For more information on updating MTU values within FSxN ONTAP follow this NetApp article.

Jumbo frame 8500 bytes MTU validation

Once both the VMkernel port in question and the FSxN broadcast domain MTU have been set to 8500 bytes, you can validate the end-to-end MTU by using vmkping. It is important to validate the entire data path when using jumbo frames to ensure that you do not experience data packet fragmentation, which can cause adverse performance effects on your storage network, such as increased latency and packet loss.

Step 1: Connect to an ESXi host within your Amazon EVS environment using SSH. You can use the example command below to test connectivity to your FSxN NFS IP address, where vmkN is the VMkernel interface you are using to connect to the FSxN file system:

vmkping -I vmkN FSxN_NFS_IP -s 8472 -d -c 20

For additional context, the -d switch ensures that the data packet does not fragment across the data path and the -c switch is the number of pings sent in the request. We are sending 8472 bytes as the packet size in the -s switch, as there is an overhead of 28 bytes in the ping header, which totals our maximum of 8500.

Repeat this step for each ESXi host within your Amazon EVS environment.

Figure 13. ESXi SSH view showing successful vmkping of 8472 bytes to the FSxN NFS IP address

If the command returns unsuccessful on any of your ESXi hosts e.g. sendto() failed (Message too long), and thus data packets are being fragmented, ensure that the MTU settings on the VMkernel port you are using, and the FSxN broadcast domain have been set correctly and the packet overhead for ping has been considered in the packet size, then re-run the test.

Result

You will have successfully created an FSxN file system and volume, connected and mounted it to all selected hosts within your EVS environment using NetApp best practices. The datastore can be used to increase the storage footprint of the environment without having to add any new hosts to the cluster. Additional datastores can be added as needed but will need nConnect performance tuning commands performed each time.

If any new ESXi hosts get added into the Amazon EVS environment after this has been completed, these steps will need to be performed on that new ESXi host only, and any existing FSxN datastores will need to be manually mounted.