Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

Why ecs-agent not supporting GPU on amazon/Deep Learning Base OSS Nvidia Driver GPU AMI (Amazon Linux 2023) 20241104

0

Nvidia drivers and toolkit available and verified with sudo docker run --rm --gpus all nvidia/cuda:11.2.2-cudnn8-runtime-ubuntu20.04 nvidia-smi > all good

But when I add ECS_ENABLE_GPU_SUPPORT=true, ecs-agent exits with 2

Running on Instance type: g4dn.xlarge

[ec2-user@ip-172-31-18-35 ~]$ systemctl status ecs.service ● ecs.service - Amazon Elastic Container Service - container agent Loaded: loaded (/usr/lib/systemd/system/ecs.service; enabled; preset: disabled) Active: activating (auto-restart) (Result: exit-code) since Fri 2024-11-15 09:53:10 UTC; 2s ago Docs: https://aws.amazon.com/documentation/ecs/ Process: 4453 ExecStartPre=/bin/bash -c if [ $(/usr/bin/systemctl is-active docker) != "active" ]; then exit 1; fi (code=exited, status=0/SUCCESS) Process: 4455 ExecStartPre=/usr/libexec/amazon-ecs-init pre-start (code=exited, status=2) Process: 4469 ExecStopPost=/usr/libexec/amazon-ecs-init post-stop (code=exited, status=0/SUCCESS) CPU: 44ms

Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal systemd[1]: ecs.service: Control process exited, code=exited, status=2/INVALIDARGUMENT ░░ Subject: Unit process exited ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ An ExecStartPre= process belonging to unit ecs.service has exited. ░░ ░░ The process' exit code is 'exited' and its exit status is 2. Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal amazon-ecs-init[4531]: level=info time=2024-11-15T09:53:30Z msg="post-stop" Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal amazon-ecs-init[4531]: level=info time=2024-11-15T09:53:30Z msg="Cleaning up the credentials endpoint setup for Amazon Elastic Container Service Agent" Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal amazon-ecs-init[4531]: level=error time=2024-11-15T09:53:30Z msg="Error performing action 'delete' for iptables route: exit status 1; raw output: iptabl> Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal amazon-ecs-init[4531]: level=error time=2024-11-15T09:53:30Z msg="Error performing action 'delete' for iptables route: exit status 1; raw output: iptabl> Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal amazon-ecs-init[4531]: level=error time=2024-11-15T09:53:30Z msg="Error performing action 'delete' for iptables route: exit status 1; raw output: iptabl> Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal amazon-ecs-init[4531]: level=error time=2024-11-15T09:53:30Z msg="Error performing action 'delete' for iptables route: exit status 1; raw output: iptabl> Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal systemd[1]: ecs.service: Failed with result 'exit-code'. ░░ Subject: Unit failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ The unit ecs.service has entered the 'failed' state with result 'exit-code'. Nov 15 09:53:30 ip-172-31-18-35.eu-west-1.compute.internal systemd[1]: Failed to start ecs.service - Amazon Elastic Container Service - container agent. ░░ Subject: A start job for unit ecs.service has failed ░░ Defined-By: systemd ░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel ░░ ░░ A start job for unit ecs.service has finished with a failure. ░░ ░░ The job identifier is 2070 and the job result is failed.

Topics
Containers
Tags
Amazon Elastic Container Service
Language
English

asked 2 years ago535 views

1 Answer
0

The issue you're experiencing with the ECS agent not supporting GPU on the Amazon Deep Learning Base OSS Nvidia Driver GPU AMI (Amazon Linux 2023) is likely due to compatibility issues between the ECS agent and the specific AMI or GPU configuration.

Several factors could be contributing to this problem:

  1. AMI Compatibility: The ECS GPU-optimized AMI might not be fully compatible with the Amazon Deep Learning Base OSS Nvidia Driver GPU AMI you're using. ECS has specific requirements for GPU support, and not all AMIs are automatically compatible.

  2. NVIDIA Driver Version: The ECS agent may be expecting a specific version of the NVIDIA drivers. While your test with the CUDA container was successful, the ECS agent might be looking for different driver configurations.

  3. ECS Agent Version: The version of the ECS agent installed on your AMI might not be compatible with the GPU support feature you're trying to enable.

  4. Configuration Issues: The ECS_ENABLE_GPU_SUPPORT=true flag alone might not be sufficient. There could be additional configuration steps required for your specific AMI and GPU setup.

  5. AMI Version: The p5 instance family, for example, is only supported on version 20230929 and later of the Amazon ECS GPU-optimized AMI. Your AMI might not meet the version requirements for GPU support.

To resolve this issue, you could try the following:

  1. Use the official Amazon ECS-optimized AMI for GPU workloads instead of the Deep Learning AMI.
  2. Ensure that all required environment variables for the NVIDIA container runtime are properly set.
  3. Check if your AMI version is compatible with the GPU instance type you're using.
  4. Review the ECS agent logs for more detailed error messages that could provide insight into why it's failing to start with GPU support enabled.
  5. Consider updating the ECS agent to the latest version compatible with your setup.

If the issue persists, you may need to contact AWS support for further assistance, as there might be specific incompatibilities between the ECS agent and the custom AMI you're using.
Sources
Amazon ECS task definitions for GPU workloads - Amazon Elastic Container Service
Use a GPU workload AMI - AWS Batch
Launch a GPU container instance for Amazon ECS - Amazon Elastic Container Service

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content