CannotPullContainerError: no space left on device


Since today I have been seeing several failures on batch jobs due to the following error:

CannotPullContainerError: failed to register layer: Error processing tar file(exit status 1): write /opt/conda/lib/ no space left on device

I am using the standard AWS ECS AMI, and my compute environment is set up to use 'm4 family' so I would expect it to choose an instance that has enough resources.

Is there anything I can do to prevent this?

asked 4 years ago2201 views
2 Answers


In order to fix this issue, you need to update the dm.base value in the docker configuration. Please refer this article to achieve the same.

answered 4 years ago
profile picture
reviewed 16 days ago

Thanks Anish, that helped. For anyone also having trouble with this, you need to

  1. increase the size of the disk available to docker (/dev/xvdcz) in the launch template
  2. increase the disk quota available to each container in the launch template, I used:
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==BOUNDARY=="

Content-Type: text/cloud-boothook; charset="us-ascii"
cloud-init-per once docker_options echo 'OPTIONS="${OPTIONS} --storage-opt dm.basesize=30G"' >> /etc/sysconfig/docker

Content-Type: text/x-shellscript; charset="us-ascii"

# more aggressive settings for cleaning up old tasks
echo ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=1m >> /etc/ecs/ecs.config
echo ECS_IMAGE_CLEANUP_INTERVAL=1m >> /etc/ecs/ecs.config
echo ECS_IMAGE_MINIMUM_CLEANUP_AGE=1m >> /etc/ecs/ecs.config

answered 4 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions