NVidia driver mismatch with sagemaker triton inference container

Question

I'm in the process of updating dependnecies of a sagemaker pipeline that's been in production for some time. I've run into an issue with the inference container, I bumped the image to `763104351884.dkr.ecr.ap-southeast-2.amazonaws.com/sagemaker-tritonserver:25.04-py3` and when I try and deploy I get

```
NVIDIA Release 25.04 (build )
Triton Server Version 2.57.0
Copyright (c) 2018-2025, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.
GOVERNING TERMS: The software and materials are governed by the NVIDIA Software License Agreement
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/)
and the Product-Specific Terms for NVIDIA AI Products
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/).
ERROR: This container was built for NVIDIA Driver Release 575.51 or later, but
       version 470.256.02 was detected and compatibility mode is UNAVAILABLE.
```

Then everything else fails as no GPUs are detected.

I didn't change the instance type - it was `ml.g4dn.4xlarge` - is the issue because certain instance types don't have the latest nvidia drivers (understandable if the GPU doesn't support the latest features) or is the driver part of the container (in which case triton sagemaker-tritonserver:25.04-py3 may have an issue?)

Assuming it's the former - is there a capabilities matrix somewhere showing what nvidia drivers are available by instance type so I can work out what instance and/or what version of triton to deploy? (I intend on testing with a more modern instance type, i.e. ml.g6.xlarge but need to get our admin to update quota and they're not in my timezone)

re:Post Agent · Answer

The issue you're encountering is due to a mismatch between the NVIDIA driver version required by your container and the one available on your instance. The Triton container version 25.04 was built for NVIDIA Driver Release 575.51 or later, but your ml.g4dn.4xlarge instance has driver version 470.256.02.

This is not an issue with the container itself, but rather a compatibility problem between the container's requirements and the driver available on your instance type. The driver is part of the instance's AMI, not the container.

You have several options to resolve this:

1. Use a newer instance type like the ml.g6.xlarge you mentioned, which likely has more recent drivers.

2. Specify a compatible inference AMI version when deploying your endpoint. SageMaker offers different AMI versions with different NVIDIA driver versions:
- al2-ami-sagemaker-inference-gpu-2 or al2-ami-sagemaker-inference-gpu-2-1: NVIDIA driver version 535, CUDA 12.2
- al2-ami-sagemaker-inference-gpu-3-1: NVIDIA driver version 550, CUDA 12.4

3. Use an older version of the Triton container that's compatible with the driver version on your instance.

4. Install the CUDA Compatibility Package in your container, which can help bridge compatibility between different driver versions.

If you continue to use the ml.g4dn.4xlarge instance, you'll need to either downgrade your container version or implement the CUDA Compatibility Package solution. For newer instances like g6, you'll likely have access to more recent drivers that are compatible with your container.

When you encounter errors like "CannotStartContainerError" or NVIDIA initialization errors, it's typically due to this kind of driver/CUDA version mismatch.  
**Sources**  
[Best practices to minimize interruptions during GPU driver upgrades - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/inference-gpu-drivers.html)  
[Updating inference containers to comply with the NVIDIA Container Toolkit - Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/latest/dg/container-nvidia-compliance.html)  
[ProductionVariant - Amazon SageMaker](https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_ProductionVariant.html)

NVidia driver mismatch with sagemaker triton inference container

Add your answer

Relevant content