DCV server taking really long time to connect to EC2 metadata servcice.

Question

See this log:

```
2025-04-22 17:24:05,590435 [   937:937   ] INFO  server - Starting DCV server version 2022.2 (r13907) - 0c005710fc2d84357a4e13e33fc6061c37ab8e15
2025-04-22 17:24:05,590562 [   937:937   ] INFO  system-info - /sys/hypervisor/uuid does not exist
2025-04-22 17:24:10,593661 [   937:937   ] INFO  aws - Cannot connect to the EC2 instance metadata service (retry-1 within 1 sec): Socket I/O timed out
2025-04-22 17:24:16,597676 [   937:937   ] INFO  aws - Cannot connect to the EC2 instance metadata service (retry-2 within 1 sec): Socket I/O timed out
2025-04-22 17:24:22,601691 [   937:937   ] INFO  aws - Cannot connect to the EC2 instance metadata service (retry-3 within 1 sec): Socket I/O timed out
2025-04-22 17:24:28,605652 [   937:937   ] INFO  aws - Cannot connect to the EC2 instance metadata service (retry-4 within 1 sec): Socket I/O timed out
[...........]
2025-04-22 17:41:57,309750 [   937:937   ] INFO  aws - Cannot connect to the EC2 instance metadata service (retry-59 within 1 sec): Socket I/O timed out
2025-04-22 17:42:03,313700 [   937:937   ] INFO  aws - Could not fetch the metadata service token: Socket I/O timed out
2025-04-22 17:42:03,314943 [   937:937   ] INFO  server - DCV server running on OS: Linux (5.15.0-1077-aws #84~20.04.1-Ubuntu SMP Mon Jan 20 22:14:54 UTC 2025), arch: x86_64 - Number of processors: 4
2025-04-22 17:42:03,314971 [   937:937   ] INFO  server - Instance ID: i-...
2025-04-22 17:42:03,314991 [   937:937   ] INFO  server - Detected EC2 instance with type 'g4dn.xlarge' on AWS region 'us-east-1'
2025-04-22 17:42:03,315069 [   937:937   ] INFO  license-manager - Creating EC2 license manager
2025-04-22 17:42:03,315148 [   937:937   ] INFO  license-manager - Initializing EC2 license manager with IID: 'b3abb9b36bbb91047590e136064b8675'
2025-04-22 17:42:03,315396 [   937:937   ] INFO  extensions - Looking for modules in '/usr/lib/x86_64-linux-gnu/dcv/modules'
2025-04-22 17:42:03,316187 [   937:937   ] INFO  DCV - DCP: {cpu-manufacturer:Intel,instruction-set:Avx2}
2025-04-22 17:42:03,318171 [   937:937   ] WARN  VAAPI:display - libva.so.2 and libva-drm.so.2 not available, vaapi disabled
2025-04-22 17:42:03,331828 [   937:937   ] INFO  DCV - DCP: {cpu-manufacturer:Intel,instruction-set:Avx2}
2025-04-22 17:42:03,332414 [   937:937   ] INFO  DCV - DCP: {cpu-manufacturer:Intel,instruction-set:Avx2}
2025-04-22 17:42:03,337020 [   937:937   ] INFO  DCV - DCP: {cpu-manufacturer:Intel,instruction-set:Avx2}
2025-04-22 17:42:03,337156 [   937:937   ] INFO  server - No metrics reporter available, metrics will be ignored
2025-04-22 17:42:03,339145 [   937:937   ] INFO  certificate-loader - User certificate not found.
2025-04-22 17:42:03,339895 [   937:937   ] INFO  certificate-loader - Generating self-signed certificate for hostname: eac28c90c3a2 (ip: 172.17.0.2)
2025-04-22 17:42:03,484702 [   937:937   ] INFO  certificate-loader - Loaded server certificate from /var/lib/dcv/.config/NICE/dcv/dcv.pem (key: /var/lib/dcv/.config/NICE/dcv/dcv.key)
2025-04-22 17:42:03,484744 [   937:937   ] INFO  certificate-loader - Server certificate fingerprint (SHA-1): ...
2025-04-22 17:42:03,484849 [   937:937   ] INFO  printer-manager - Loading Printer Manager extension
2025-04-22 17:42:03,484861 [   937:937   ] INFO  printer-manager - Extension name: cups-manager
2025-04-22 17:42:03,484981 [   937:937   ] INFO  Cups:cups-manager - Received remove all printers request
2025-04-22 17:42:03,485028 [   937:1976  ] INFO  Cups:cups-manager - Starting worker
2025-04-22 17:42:03,485071 [   937:1976  ] INFO  Cups:cups-manager - Worker: Removing all virtual printers
2025-04-22 17:42:03,485241 [   937:937   ] INFO  server - Using 'none' authentication method
2025-04-22 17:42:03,485750 [   937:937   ] INFO  http-static-files - 	HSTS enabled with parameters : max-age=31536000; includeSubDomains
2025-04-22 17:42:03,485771 [   937:937   ] INFO  http-static-files - No extra HTTP headers specified
2025-04-22 17:42:03,485799 [   937:937   ] INFO  http-version - Create HTTP version handler: 2022.2-13907
2025-04-22 17:42:03,486003 [   937:937   ] INFO  http-service - Successfully listening on TCP socket bound to '0.0.0.0'
2025-04-22 17:42:03,486095 [   937:937   ] INFO  http-service - Successfully listening on TCP socket bound to '::'
2025-04-22 17:42:03,486113 [   937:937   ] INFO  http-service - HTTPS service started on 2 endpoints (default port: 8443, URL path: /, web root: /usr/share/dcv/www, allowed host pattern: ^.+$, allowed origin pattern: ^https://.+$, X-Frame-Options: DENY)
2025-04-22 17:42:03,486175 [   937:937   ] INFO  server - Using backend-authentication methods 'user,path' with timeout 5000
2025-04-22 17:42:03,486315 [   937:937   ] INFO  backend-service - Started backend service: /com/nicesoftware/dcv/server
2025-04-22 17:42:03,486375 [   937:937   ] INFO  system-resource-monitor - Start monitoring system resources
2025-04-22 17:42:03,486475 [   937:937   ] INFO  backend-service - Started backend service: /com/nicesoftware/dcv/rpc
2025-04-22 17:42:03,486547 [   937:937   ] INFO  server - USB driver is not installed: Failed to open /dev/eveusb: No such file or directory. If needed, please run /usr/bin/dcvusbdriverinstaller to install it
2025-04-22 17:42:03,492849 [   937:937   ] INFO  rpc-handler - Subscribe system signals request 'subscribe-sys-signal:0' from rpc client '0x7fb044016050'
2025-04-22 17:42:03,493038 [   937:937   ] INFO  rpc-handler - Subscribe application signals request 'subscribe-app-signals:1' from rpc client '0x7fb044016050'
```
It is taking 15-20 minutes to connect to the EC2 metadata service to verify the license, which blocks me being able to use DCV.

A little about my setup:
I am running DCV on Ubuntu in a Docker container, using this run command:

```
docker run -d --rm --gpus all --privileged \
    -p "$PORT:8443" \
    --name $CONTAINER_NAME \
    --add-host=metadata.aws.internal:169.254.169.254 \
    --add-host=metadata:169.254.169.254 \
    "$CONTAINER_IMAGE"
```

I can use curl to access IMDS in the terminal, it is only DCV that seems to have the issue.

Here is a summary of the troubleshooting I've done:
* Curl worked inside the container to access the EC2 metadata service.
* nsenter confirmed that the dcvserver process could also access the EC2 metadata service.
* IAM role was verified as attached to the instance and credentials were accessible via the metadata service.
* IAM role included AmazonEC2ContainerRegistryFullAccess, AmazonEC2ContainerServiceforEC2Role, AmazonDynamoDBFullAccess, AmazonS3FullAccess, and AWSLambda_FullAccess.
* Container startup script included debug logging to confirm metadata access and public IP detection.
* Attempted to restart dcvserver using systemctl, same issue

Would really appreciate any help on this one,
Cheers,
Tom

Tom · Accepted Answer

If anyone comes back to this thread later, the issue is the number of hops, you can fix locally by running:
```
aws ec2 modify-instance-metadata-options \
    --instance-id i-123 \
    --http-put-response-hop-limit 2 \
    --region us-east-1
```

re:Post Agent · Answer

The issue you're experiencing with DCV taking a long time to connect to the EC2 metadata service is likely related to how Docker is handling network requests to the metadata service IP address (169.254.169.254).

While you've added host entries for the metadata service, Docker's networking setup may still be interfering with DCV's ability to reach the metadata service efficiently. Here are a few suggestions to troubleshoot and potentially resolve the issue:

1. Instead of using --add-host, try using the --network host option in your Docker run command. This will allow the container to use the host's network stack directly, which should provide unimpeded access to the EC2 metadata service.

2. If using --network host is not feasible for your setup, you can try creating a custom Docker network that includes a route to the EC2 metadata service IP. Create a network with:

docker network create --driver bridge --subnet=172.18.0.0/16 metadata-net
   
   Then add --network metadata-net to your Docker run command.

3. Ensure that the security group and network ACLs associated with your EC2 instance allow outbound traffic to the metadata service IP (169.254.169.254) on port 80.

4. Check if there are any VPC endpoints or NAT gateways that might be interfering with the metadata service access.

5. Verify that the IAM role attached to your EC2 instance has the necessary permissions to access the DCV license.

6. As a temporary workaround, you could try increasing the timeout for metadata service connections in the DCV configuration, if possible.

If these suggestions don't resolve the issue, you may need to reach out to AWS support for further assistance, as there could be region-specific or account-specific factors affecting the metadata service access for DCV licensing.  
**Sources**  
[DCV - No license available from eu-south-2 region | AWS re:Post](https://repost.aws/questions/QUgiD6H8tYS9ygiCsv7Mn2og/dcv-no-license-available-from-eu-south-2-region)  
[Troubleshoot metadata errors in my ECS Fargate tasks | AWS re:Post](https://repost.aws/knowledge-center/ecs-fargate-metadata-errors)

DCV server taking really long time to connect to EC2 metadata servcice.

Relevant content