Moving VMware HCX Service Meshes from VPN to Direct Connect

One of my customers are in the journey of migrating the workloads from On-Premise Datacenter to VMware Cloud on AWS. They have a 8 node VMC SDDC brought up and they are connected via a VPN tunnel over the public Internet. They also have HCX deployed on premise with multiple stretched networks and two HCX Service Meshes.

The existing service meshes was created over the HCX VPN tunnel for the workload migration. Due to the fact that the customer is now planning for the mass VM migration from On-Premise to VMC, they decided to go with Direct Connect (AWS Direct Connect)

In this blog spot, I share the steps we performed.

Architecture

We have setup the Direct Connect between On-Prem and AWS Datacenter and the connections are made available in AWS network account. Then created the Transit Virtual interfaces and associated with the Direct connect gateway (https://docs.aws.amazon.com/directconnect/latest/UserGuide/direct-connect-gateways-intro.html ). Then attach the Direct Connect Gateway to an SDDC group.

The high-level architecture looks like:

Step 1 (Configuration in AWS Network account)

Once the links from network backbone is setup, the links becomes visible In the AWS network account as below

Step 2 : (Configuration in VMC Console)

Now the VMC should attach the VIF to the AWS Direct connect gateway using the Private. Once the links are up the status of the Direct connect shows as below.

Make sure the direct connect is Connected to VMC SDDC group as shown in the picture below

Post establishing the Direct Connection from On-premise to VMC, the next step to create the Service mesh over the newly created DX links.

Step 3 : (Configuration in VMware HCX)

Pre-requisites and health check:

1. In the VMC SDDC - Direct connect tab, make sure the routes are advertised for vSphere management, vCenter and the HCX management network segments.
2. Update the management firewall rules in VMC SDDC to have the new Direct Connect segments (previously it has the network segments for VPN Connection)
3. Test the connectivity (ping) from VMC SDDC to the on-premise vcenter segments that is being advertised over the Direct Connect to verify the on-premise vSphere management can communicate with VMC SDDC Management appliances
4. Now the connections are up and ensure there are no live migrations or replications are in progress in HCX.
5. Change the HCX FQDN Resolution address

• Log in to VMware Cloud Services at https://vmc.vmware.com.
• Click Inventory > SDDCs, then pick an SDDC card and click VIEW DETAILS.
• Navigate to the Settings tab of your SDDC.
• Expand HCX FQDN, and click Edit.
• Under Resolution Address select the Private IP address ( new after DX setup) and click SAVE.

6. Verify HCX Connector & HCX Cloud pairing is healthy. The existing HCX site pairing should work without any issues. If not, we need ot update the site pairing. 7.Create the Direct Connect Network Profile in VMware HCX in VMC.

• 1. Navigate to the SDDC tab and login using the cloudadmin@vmc.local user or login with the enterprise admin credentials.
• 2. Navigate to the Infrastructure and Interconnect then the Network profiles
• 3. Create a new DirectConnectNetwork network profile
• 4. Add the IP range, prefix length, and gateway to the network profile and click SAVE. (this is the new IP range for the Direct Connect setup and it should be unique without any overlaps in VMC or On-prem)
• 5. Check back that the new network segment is advertised over the Direct connect in VMC SDDC Console ->Networking & Security->System->Direct Connect.  The subnet should be visible under Advertised BGP Routes.
• 6. Update the on-premise firewalls are allowing the new HCX IP Range to communicate with the vSphere Management network on premise.

8. Redeploy HCX Service Mesh from on premise HCX Connector Manager 9.Update new compute firewall rules in VMC SDDC to referred to the prior VPN connection to the Direct Connect.

Important notes

1. Make sure there are no ongoing replications or migrations in the HCX.
2. Make sure no networks are extended over the Service mesh which we are updating
3. Redeploy the service mesh on-prem as needed