Using Launch Template with custom AMI for AWS Batch

Question

We created an AMI with a preloaded docker image and ECS agent configurations (under /etc/ecs/ecs.config) in hopes to bring our AWS Batch execution time down. Although the pull time has been eliminated, the job execution time has doubled (from our prod jobs that still pull the docker image).

Has anyone experienced this issue or have guidance on how to debug?

Answer

I would like to have more context of the issue.

Like, for how long your job run generally? Also, caching container image should not hamper the execution time. So to debug such issue, i would suggest you add timestamp entries in logs to check which part of application code is taking more time to execute.

And how much image pull time is reduced after caching image? you can confirm the same via check corresponding ecs task details which will have image pull time details.

Answer

For now we have one EC2 instance always running to mitigate the cold start problem (EC2 instance spin-up + docker image pull).
Jobs run for ~3:30min, ideally image pull time is eliminated after caching. Also, we've reduced the image size from 16GB to 4GB.

Using Launch Template with custom AMI for AWS Batch

Relevant content