ECS Agent detects GPU UUID but reports integerValue=0 on g6f.2xlarge instance with 0.25 NVIDIA L4

I’m running Amazon ECS with a g6f.2xlarge. Since there’s no GPU-optimized ECS AMI for this instance family yet, I have included GRID driver and ecs agent installation in my user data for my launch template for now with the base as Amazon Linux 2023 6.1

Installed NVIDIA GRID driver (18.4) from the AWS S3 bucket → nvidia-smi works correctly on the host. Installed Docker + NVIDIA container runtime (nvidia-container-runtime, nvidia-container-toolkit) and configured /etc/docker/daemon.json. ECS agent runs in a Docker container with runtime set to nvidia.

I can run GPU containers manually with:

docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi

and it shows 6GB VRAM in nvidia-smi as expected.

ECS agent logs show the GPU UUID is detected:

GPU:{... StringSetValue:[GPU-fe0ee694-992c-11f0-b284-3159a5beec23]}

But when I describe the container instance, ECS reports:

{
  "name": "GPU",
  "type": "STRINGSET",
  "integerValue": 0,
  "stringSetValue": [
    "GPU-fe0ee694-992c-11f0-b284-3159a5beec23"
  ]
}

So the GPU count (integerValue) stays 0, which prevents ECS from scheduling GPU tasks.

What I’ve verified:

• /dev/nvidia* devices exist on the host.
• ECS agent is running with --runtime=nvidia.
• Toolkit and runtime installed/configured correctly.
• No errors in ECS agent logs.

Question: Is this a limitation in the current ECS agent with the new fractional GPU instances where it detects the device but doesn’t mark it as allocatable (as it is technically not a whole GPU)? If so, is there a workaround (e.g., forcing GPU count, custom task/device mappings) or a newer agent version that can solve my issue?

Below is my current user data used in the launch template. I have also tried adding --device /dev/nvidia* in the ExecStart directly but that didn't work either

#!/bin/bash
set -x
exec > >(tee /var/log/user-data.log) 2>&1

# === NVIDIA GRID Driver Installation ===
dnf update -y
dnf install -y gcc make
dnf install -y kernel-devel kernel-modules-extra
aws s3 cp --recursive s3://ec2-linux-nvidia-drivers/grid-18.4/ .
chmod +x NVIDIA-Linux-x86_64*.run
/bin/sh ./NVIDIA-Linux-x86_64*.run --silent --accept-license --no-questions

# === ECS Agent Setup ===
# Add NVIDIA repo manually
tee /etc/yum.repos.d/nvidia-container-toolkit.repo <<'EOF'
[nvidia-container-toolkit]
name=NVIDIA Container Toolkit
baseurl=https://nvidia.github.io/libnvidia-container/stable/rpm/x86_64
enabled=1
gpgcheck=1
gpgkey=https://nvidia.github.io/libnvidia-container/gpgkey
EOF

# Install Docker
dnf install -y docker
systemctl enable --now docker

# Install NVIDIA Conatiner Toolkit
dnf clean expire-cache
dnf install -y nvidia-container-toolkit
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

# Create ECS config
mkdir -p /etc/ecs
cat <<'EOF' > /etc/ecs/ecs.config
ECS_CLUSTER=nfl_prod
ECS_ENABLE_CONTAINER_METADATA=true
ECS_ENABLE_GPU_SUPPORT=true
ECS_ENABLE_GPU=true
ECS_AVAILABLE_GPU_DEVICES=all
EOF

# Ensure required directories exist
mkdir -p /var/log/ecs /var/lib/ecs/data

# Create systemd unit for ECS Agent
cat <<'EOF' > /etc/systemd/system/ecs.service
[Unit]
Description=Amazon ECS agent
After=docker.service
Requires=docker.service

[Service]
Restart=always
ExecStartPre=-/usr/bin/docker rm -f ecs-agent
ExecStartPre=/bin/mkdir -p /var/log/ecs /var/lib/ecs/data
ExecStartPre=/usr/bin/docker pull public.ecr.aws/ecs/amazon-ecs-agent:latest
ExecStart=/usr/bin/docker run --name ecs-agent \
  --runtime=nvidia \
  --privileged --network=host \
  -v /var/run:/var/run \
  -v /var/log/ecs:/log \
  -v /var/lib/ecs/data:/data \
  -v /sys/fs/cgroup:/sys/fs/cgroup \
  -v /run/systemd:/run/systemd \
  -e ECS_LOGFILE=/log/ecs-agent.log \
  -e ECS_DATADIR=/data \
  -e ECS_CLUSTER=nfl_prod \
  -e ECS_ENABLE_GPU=true \
  public.ecr.aws/ecs/amazon-ecs-agent:latest

[Install]
WantedBy=multi-user.target
EOF

# Reload systemd and enable ECS service
systemctl daemon-reload
systemctl enable --now ecs