Using Amazon EC2 as an External Backup Repository for Amazon EVS VCF Management Appliances

Background

The Amazon Elastic VMware Service (Amazon EVS), deploys VMware Cloud Foundation (VCF) with essential management appliances including vCenter Server, NSX Manager, and SDDC Manager. These appliances are critical to the operation and management of your EVS environment, handling everything from virtual machine orchestration to software-defined networking and lifecycle management.

By default, the SDDC Manager and NSX Manager appliances come preconfigured with backup repositories that point to local storage on the SDDC Manager appliance, however, the vCenter appliance does not have backups configured at all. While having some backup capability out of the box is helpful, this configuration creates operational challenges in two key scenarios:

1. Appliance Recovery: If the SDDC Manager appliance becomes corrupted or experiences a failure, recovering NSX Manager and SDDC Manager backups stored on that same appliance becomes problematic or impossible. This self-referential backup configuration limits your ability to perform a clean restoration. Additionally, vCenter has no backup protection by default, leaving a critical management component vulnerable.

2. VCF Upgrades: During VMware Cloud Foundation upgrades, having backups stored locally on the appliances being upgraded introduces risk. In fact, you cannot proceed with a VCF environment upgrade until the backup repositories have been reconfigured to point to external storage rather than local repositories, and vCenter backups have been configured. This is a hard requirement that blocks the upgrade process until resolved.

The solution is to redirect backup storage to an external repository, separating your backup data from the appliances themselves. This architectural best practice ensures that your backup data remains accessible and protected, independent of the state of your management appliances.

This article demonstrates how to quickly deploy an Amazon EC2 instance as a dedicated SFTP server, and reconfigure your SDDC Manager, NSX Manager and vCenter to store all backups on this external repository, providing a more resilient and operationally sound backup architecture.

Situation

A customer needed to upgrade their Amazon EVS environment to VCF 5.2.2 to take advantage of the latest features, security patches, and performance improvements. However, when they initiated the upgrade process, they discovered they were unable to proceed because their VCF appliance backups were still pointing to local repositories.

This is a common scenario for customers operating with the default EVS backup configuration. The VCF upgrade pre-flight checks identified the local backup configuration as a blocking issue, preventing the upgrade from moving forward. Without an external backup repository in place, the customer was stuck—unable to upgrade their environment until this prerequisite was satisfied.

The customer needed to quickly deploy a new backup server for VCF appliance backups outside of their Amazon EVS environment, and reconfigure all management appliances to use this external repository before the upgrade could proceed. This situation highlights the importance of establishing external backup storage early in your EVS operational lifecycle, rather than waiting until an upgrade forces the issue.

Tasks

1. Deploy an Amazon EC2 instance within the Amazon EVS service access subnet, using the provided user data template to automatically configure the SFTP server, backup user, group and openssh configuration upon deployment.

2. Update Backup configuration settings within SDDC Manager, NSX Manager and vCenter to point to new EC2 instance

3. Perform a Backup of SDDC Manager, NSX Managers and vCenter appliances.

Prerequisites

1. Create an IAM role for EC2, with the following IAM policy permissions attached:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:TagResource"
      ],
      "Resource": "arn:aws:secretsmanager:*:*:secret:*"
    }
  ]
}

Permission details:

• secretsmanager:CreateSecret - Required to create the new secret in Secrets Manager for storing the sftp user account password

• secretsmanager:TagResource - Optional, but recommended if you want to add tags to the secret for organization and cost tracking

Actions

Task 1: Deploy Amazon Linux EC2 Instance

1. Using the AWS console, navigate to EC2 and choose Launch Instance.
2. Next, provide a Name and Tag as required.
3. Choose Amazon Linux from the Application and OS Images (Amazon Machine Image) selection, and select Amazon Linux 2023 kernel-6.1 AMI as the AMI image.

Note: This guide has user data defined in a cloud-config format for an Amazon Linux deployment, and is validated to work with Amazon Linux 2023 kernel 6.1. You can take what is here and change for any other OS type, such as Ubuntu, as required.

4. Select an appropriate Instance Type to run your SFTP backup server. In our example, we used a t3.medium EC2 instance.

Note: There will be cost implications for whichever EC2 instance type you choose. Please review EC2 pricing information here for more detailed information.

5. Next, create a new or choose an existing Key pair.

6. Next, under Network settings, select Edit. Ensure you select your Amazon EVS VPC and Amazon EVS service access subnet. Ensure you select Disable for Auto-assign Public IP. Create a new or select an existing Security Group, per your security requirements.

Note: Whichever Security Group you select, you will need to ensure to allow SSH/SFTP traffic on TCP 22 to the IP addresses of your SDDC Manager, NSX Managers and vCenter Server from your EC2 instance.

7. Under Configure storage, set the size of the volume appropriately for the amount of backup file storage you wish to keep. EBS volumes can be resized as needed if you need to increase later. Choose the storage type as per your requirements i.e. gp3, st1, etc. In our example we used 200GB gp3.

Note: There will be cost implications for how much storage you provision for whichever EBS volume type you choose. Please review EBS pricing information here for more detailed information.

8. Next, hit the down arrow to expand Advanced details. There are 2 changes you need to make here: First, select the correct IAM instance profile, with the IAM policy per the prerequisites section of this guide. Secondly, scroll down to User data - optional and paste in the following YAML syntax, updating the REGION under runcmd as needed:

#cloud-config
package_update: true # Refreshes available packages and versions
package_upgrade: true # Installs any available package updates to latest versions
groups:
 - sftp # Create local sftp group
users:
 - name: sddc_backup # Create backup user
   groups: sftp # Add backup user to sftp group
   shell: /bin/bash # Set shell to bash
   lock_passwd: false # Do not lock users password
write_files:
 - path: /etc/ssh/sshd_config.d/sftp_group.conf # Create ssh.d config for sftp users group
   content: |
      Match group sftp  #  Apply these settings to members of the sftp group
        ChrootDirectory /backups # Set chroot directory for backup user to /backups
        X11Forwarding no # Disables running graphical applications on the server
        AllowTcpForwarding no # Disables ability to route traffic through the server
        ForceCommand internal-sftp  # Restricts the backup user to SFTP-only server access
   permissions: "0644" # Set permissions of the sftp_group.conf file
 - path: /etc/ssh/sshd_config.d/sddc_backup_user.conf # Create ssh.d config for backup user 
   content: |
      Match User sddc_backup
        PasswordAuthentication yes # Enable password authentication for the backup user
   permissions: "0644" # Set permissions of the sddc_backup_user.conf file
runcmd:
  - INSTANCE_ID=$(ec2-metadata --instance-id | cut -d " " -f 2)  # Capture the instance id from EC2 metadata 
  - PASSWORD=$(openssl rand -base64 32) # Generate random 32-character password
  - REGION="us-east-1" # CHANGE THIS REGION TO MATCH WHERE YOU WISH TO SAVE YOUR AWS SECRET MANAGER SECRET
  - echo "sddc_backup:$PASSWORD" | chpasswd # Set the generated password for sddc_backup user
  - aws secretsmanager create-secret --name "$INSTANCE_ID-sddc_backup_user" --secret-string "{\"username\":\"sddc_backup\",\"password\":\"$PASSWORD\",\"instance_id\":\"$INSTANCE_ID\"}" --description "AmazonLinux SFTP user password for EC2 instance $INSTANCE_ID" --region "$REGION"  #  Creates a secret within AWS Secrets Manager with the randomly generated password for secure retrieval of the SFTP password credential.
  - mkdir -p /backups # Create backup directory
  - chmod 755 /backups # Set permissions on the /backups directory
  - chown root:root /backups # Set ownership of /backups directory to root account
  - mkdir /backups/evs # Create evs specific directory within /backups
  - chmod 700 /backups/evs # Set permissions on the /backups/evs directory
  - chown sddc_backup:sftp /backups/evs # Set ownership of /backups/evs directory to backup user account
  - systemctl restart sshd # Restart the SSH service to apply the new configuration

Once completed, choose Launch instance on the right hand side.

Note: It may take a few minutes when the EC2 instance is launched to fully apply the cloud-config configuration settings and for SSH/SFTP to allow connections. You can view the status of the deployment by choosing your EC2 instance from the list of instances, selecting Actions > Monitor and troubleshoot > Get system log. You can refresh this to update the log as needed.

Next you will need to grab the credentials for the SFTP user so you can authenticate from your VCF management appliances when re-configuring your backups.

9. From the AWS Console, navigate to AWS Secrets Manager. Ensure you are in the AWS Region specified in the user data YAML syntax. Search for the secret named: INSTANCE_ID-sddc_backup_user, with INSTANCE_ID being the instance id of your EC2 instance.

Figure 1: AWS Secrets Manager showing generated secret containing credentials for SFTP user

10. Choose the Secret name to open the details, and from under Secret value, select Retrieve secret value. From here you will find the credentials for the SFTP backup user.

Figure 2: AWS Secrets Manager example secret for your SFTP user.

Tasks 2 & 3: Re-configure VCF Appliance backup settings and perform backups

Now that your SFTP server is configured, you will need to update the backup settings on your SDDC Manager, NSX Manager and vCenter Server to point to it.

1. First, log into your Amazon EVS SDDC Manager with a user assigned the Admin role. Navigate to Administration > Backup > Site Settings. Select Edit. Update the settings to point to the EC2 instance private IP address, using the credentials from your AWS Secrets Manager secret. Next, select Save.

Note: When you select Save, SDDC Manager will test connectivity to your EC2 instance using the settings you define, and will advise if there are connectivity issues. If the settings save, connectivity was successfully established, else troubleshoot by double checking the settings and Security Groups to ensure the correct IP addresses are allowed to connect on TCP 22.

Figure 3: Example SDDC Manager Backup site settings, updated to point to EC2 instance.

2. Next, under Backup > SDDC Manager Configurations, select the Backup Now button to backup the SDDC Manager and validate a successful backup competes.

Figure 4: Example SDDC Manager Backup schedule settings, with on-demand backup now button.

3. Once Backup Site settings have been successfully applied, NSX Manager will automatically inherit these settings via API. It can take up to 1 hour to successfully update. To verify, log in to your Amazon EVS NSX Manager and navigate to System > Backup & Restore. You will find the configuration under Configuration & History.

Figure 5: Example NSX Manager Backup configuration, inherited from SDDC Manager to point to the EC2 instance.

Note: See Broadcom documentation here for more detail on SDDC Manager Backup configuration.

4. Next, log into your Amazon EVS vCenter Server Appliance Management Interface (VAMI). This is typically accessed on TCP port 5480 from your vCenter Server IP address e.g. https://vcenter.fqdn:5480

5. Select Backup from the left hand plane, then from the right hand plane, select Edit.

6. Enter the backup location in the sftp:// format to the private IP address of your EC2 instance e.g. sftp://ec2ip/backup. Enter your backup server credentials from your AWS Secrets Manager secret, and set your retention policy per your requirements, along with which data you wish to include in your backup set. Once complete, select Save.

Note: When select Save, vCenter will test connectivity to your EC2 instance using the settings you define and will advise if there are connectivity issues. If the settings save, connectivity was successfully established, else troubleshoot by double checking the settings and Security Groups to ensure the correct IP addresses are allowed to connect on TCP 22.

Figure 6: Example vCenter Server Backup Schedule settings, updated to point to EC2 instance.

7. Next, from the Backup Schedule overview, select Backup Now from the right hand side to validate a successful backup completes.

Figure 7: Example vCenter Server Backup Schedule, displaying settings, backup history and backup now button.

The Backups for your SDDC Manager, NSX Manager and vCenter now point to an external repository.

Conclusion

By deploying an Amazon EC2 instance as a dedicated SFTP server and reconfiguring your VCF management appliances to use this external backup repository, you have established a more resilient and operationally sound backup architecture for your Amazon EVS environment. Your SDDC Manager, NSX Manager, and vCenter backups are now stored independently from the appliances themselves, eliminating the risks associated with self-referential backup configurations.

This external backup approach delivers several key benefits:

• Appliance Recovery: Your backup data remains accessible even if a management appliance experiences corruption or failure, enabling clean restoration procedures.
• Upgrade Readiness: Your environment now meets the prerequisite requirements for VCF upgrades, allowing you to proceed with version updates without backup-related blockers.
• Operational Independence: Backup data is decoupled from the appliances being backed up, following infrastructure best practices for data protection.

With your EC2 SFTP server in place and all VCF appliance backups successfully redirected, you're now positioned to maintain your EVS environment with confidence. Whether you're planning a VCF upgrade to version 5.2.2 or simply ensuring business continuity, your backup architecture provides the foundation for reliable operations.

As a best practice, regularly verify that your backups are completing successfully and test your restoration procedures to ensure your backup strategy delivers when needed. Ensure that old backups which are no longer required are cleaned up to save on disk space.

For additional guidance on Amazon EVS backup and recovery operations, consult the Amazon EVS documentation or reach out to your AWS account team.