By using AWS re:Post, you agree to the Terms of Use

CannotPullContainerError: no space left on device


Since today I have been seeing several failures on batch jobs due to the following error:

CannotPullContainerError: failed to register layer: Error processing tar file(exit status 1): write /opt/conda/lib/ no space left on device

I am using the standard AWS ECS AMI, and my compute environment is set up to use 'm4 family' so I would expect it to choose an instance that has enough resources.

Is there anything I can do to prevent this?

asked 2 years ago488 views
2 Answers


In order to fix this issue, you need to update the dm.base value in the docker configuration. Please refer this article to achieve the same.

answered 2 years ago

Thanks Anish, that helped. For anyone also having trouble with this, you need to

  1. increase the size of the disk available to docker (/dev/xvdcz) in the launch template
  2. increase the disk quota available to each container in the launch template, I used:
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==BOUNDARY=="

Content-Type: text/cloud-boothook; charset="us-ascii"
cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=30G"' >> /etc/sysconfig/docker

Content-Type: text/x-shellscript; charset="us-ascii"

# more aggressive settings for cleaning up old tasks
echo ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=1m >> /etc/ecs/ecs.config
echo ECS_IMAGE_CLEANUP_INTERVAL=1m >> /etc/ecs/ecs.config
echo ECS_IMAGE_MINIMUM_CLEANUP_AGE=1m >> /etc/ecs/ecs.config

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions