How to run an AWS Batch job from a local docker image (loaded in custom AMI) instead of from ECR?

Question

I have problems with docker pull times in my AWS Batch jobs. I reduced my image size as much as I could and till get 2min pull.

I have created a custom AMI that contains the image already pulled and loaded into docker. When using this image from an instance that I launch myself, let's say through the console, I can see the image and use it straightaway. However, when trying to do the same inside a Bath job (using a compute environment that contains a launch template pointing to the AMI), it seems to always try to pull from ECR no matter what... and I get the following error:

"CannotPullImageManifestError: Error response from daemon: errors: denied: requested access to the resource is denied unauthorized: authentication required"

My job definition looks like this:

{
    "image": "segmentationactivity-local:latest",
    "command": [],
    "jobRoleArn": "arn:aws:iam::MMMM:role/batchJobRole",
    "ulimits": [],
    "resourceRequirements": [
        {
            "value": "4",
            "type": "VCPU"
        },
        {
            "value": "8192", 
            "type": "MEMORY"
        },
        {
            "value": "1",
            "type": "GPU"
        }
    ],
    "secrets": []
}

Does anyone know how to do this?

Thanks

re:Post Agent · Answer

Unfortunately, AWS Batch does not directly support running jobs from local Docker images pre-loaded in a custom AMI. AWS Batch is designed to work with container images stored in container registries, primarily Amazon Elastic Container Registry (ECR).

The error you're encountering suggests that AWS Batch is still trying to pull the image from a registry, even though you've specified a local image name. This is because AWS Batch expects the image to be in a registry it can access.

To address your issue with long pull times, there are a few alternative approaches you could consider:

1. Use ECR with optimizations:
   - Ensure you're using the latest ECR features like image scanning and caching to optimize pull times.
   - Consider using multi-stage builds to reduce your image size further.

2. Use a private Docker registry:
   AWS Batch now supports private Docker registries with managed Elastic Container Service (ECS) compute environments. You could set up a private registry within your VPC to potentially reduce pull times.

3. Modify your job submission process:
   Instead of trying to use a pre-loaded image, you could create a wrapper script in your custom AMI that checks for the presence of the local image, and only pulls from ECR if it's not present. This script would then be the entry point for your Batch job.

4. Use Amazon ECS container instance draining:
   This feature allows you to keep container instances running between jobs, potentially reducing the need for frequent image pulls.

While these solutions don't directly allow you to use a local image as you initially intended, they may help address the underlying issue of long pull times. If pull times remain a significant issue even after optimization, you may need to reconsider your overall architecture or reach out to AWS support for more specific guidance.

Remember, when using custom AMIs or modifying job submission processes, ensure that your approach aligns with AWS Batch's security and operational requirements.
      
**Sources**  
[AWS Batch now supports private registry on ECS compute environments](https://aws.amazon.com/about-aws/whats-new/2024/02/aws-batch-private-registry-ecs-compute-environments/)  
[Running batch jobs at scale with EC2 Spot Instances](https://aws.amazon.com/getting-started/hands-on/run-batch-jobs-at-scale-with-ec2-spot/)

How to run an AWS Batch job from a local docker image (loaded in custom AMI) instead of from ECR?

Add your answer

Relevant content