Background

I created a Windows Server 2019 AMI with VMware workstation 15 installed. Before I captured the instance into an AMI I ran InitializeInstance.ps1 -Schedule. When I shared the AMI with another account which was launched in a different VPC, we tried connecting via SSM, but the AWS console was telling us that the instance wasn't configured properly. When we checked the logs we got the following output:

Windows sysprep configuration complete.
2022/04/20 16:40:16Z: Message: Failed to add routes.. attempting it again
2022/04/20 16:42:31Z: Message: Failed to add routes.. attempting it again
2022/04/20 16:44:46Z: Message: Failed to add routes.. attempting it again
2022/04/20 16:47:02Z: Message: Failed to add routes.. attempting it again
2022/04/20 16:49:17Z: Message: Failed to add routes.. attempting it again
2022/04/20 16:51:33Z: Message: Failed to add routes.. attempting it again
2022/04/20 16:53:49Z: Message: Failed to add routes.. attempting it again
2022/04/20 16:56:05Z: Message: Failed to add routes.. attempting it again
2022/04/20 16:58:20Z: Message: Failed to add routes.. attempting it again
2022/04/20 17:00:36Z: Message: Failed to add routes.. attempting it again
2022/04/20 17:02:53Z: Message: Failed to add routes.. attempting it again
2022/04/20 17:05:09Z: Message: Failed to add routes.. attempting it again
2022/04/20 17:06:14Z: EC2LaunchTelemetry: IsTelemetryEnabled=true
2022/04/20 17:06:14Z: EC2LaunchTelemetry: IsAgentScheduledPerBoot=true
2022/04/20 17:06:14Z: EC2LaunchTelemetry: AgentCommandErrorCode=1

When I opened the EC2Launch.log file, the exception details showed the following:

Failed to add routes.. attempting it again Cannot bind argument to parameter 'Addresses' because it is null

This error is in line 103 of C:\ProgramData\Amazon\EC2-Windows\Launch\Module\Scripts\Add-Routes.ps1 (only mention of an Addresses parameter in the Add-Routes function):

[string[]]$defaultGateways = @(FilterIPAddresses -Addresses $networkAdapterConfig.DefaultIPGateway -AddressFamily $AddressFamily)

I did a bit of source code diving to discover the source of the issue, which is described in the following section.

Bug Description

InitializeInstance.ps1 calls - among other things - a function called Add-Routes defined in C:\ProgramData\Amazon\EC2-Windows\Launch\Module\Scripts\Add-Routes.ps1. This function adds the correct network routes in order to be able to reach various services via the reserved IP addresses in 169.254.169.0/24. To do so, it adds routes telling the operating system to route all of these packets through the default gateway of a primary network interface.

The Add-Routes.ps1 script determines the primary network interface by performing the following query on line 85:

$networkAdapter = Get-CimInstance -ClassName Win32_NetworkAdapter -Filter "AdapterTypeId='0' AND NetEnabled='True' AND NOT Name LIKE '%Hyper-V Virtual%' AND NOT Name LIKE '%Loopback Adapter%' AND NOT Name LIKE '%TAP-Windows Adapter%'" | Sort-Object -Property "InterfaceIndex" | Select-Object InterfaceIndex

and selecting the interface with the lowest value of InterfaceIndex. Notably, this query does not filter out values like "VMware Virtual Ethernet Adapter for VMnet 1" which is created whenever VMware workstation is installed.

When I expermented I found that the aforemention virtual ethernet adapter had a lower InterfaceIndex than the primary interface "AWS PV Network Device #0".

Since the virtual adapter does not have a default gateway, the result of trying to access it is $null which throws the exception and makes it unable to add the correct routes for the AWS services, rendering SSM inaccessible for that instance.

Solution...?

Either, don't install VMware or change the indices of the virtual network adapters. This solves my specific use-case, but doesn't solve the future problem of having any other virtual adapters that don't get filtered out by that query which might have a lower index.

Ideally the query to find a primary network interface for the default gateway should be altered to be more robust. Rather than filtering known bad values, a better solution might be to filter for known good values, thereby eliminating the need to filter out all possible virtual network adapters, which could have arbitrary names.