By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

"Cannot write: No space left on device" error even though there is storage available

0

I'm trying to install a file transfers (Aspera CLI client for Linux), which I have done multiple times in the past with no problem, and get the "tar: cannot write: No space left on device" error. I've checked using df -h command and there is plenty storage available. Filesystem Size Used Avail Use% Mounted on udev 79G 12K 79G 1% /dev tmpfs 16G 404K 16G 1% /run /dev/xvda1 50G 48G 0 100% / none 4.0K 0 4.0K 0% /sys/fs/cgroup none 5.0M 0 5.0M 0% /run/lock none 79G 0 79G 0% /run/shm none 100M 0 100M 0% /run/user overflow 1.0M 0 1.0M 0% /tmp /dev/xvdg 8.8T 7.4T 990G 89% /home/ubuntu/users /dev/xvdh 3.9T 1.9T 1.9T 50% /home/ubuntu/proc

Can you advise on how to address this? Thank you

Topics
Storage
Tags
Storage
Language
English
tu
asked 5 months ago312 views
1 Answer
0

Firstly, the df -h output in the question is a bit garbled, could you edit the question so that it's enclosed in the code tag please? Either select the text and click </> or else put three backticks before the first line of output, and the same after the last line.

But picking through the output in its current form, it can be seen that the root filesystem is full /dev/xvda1 50G 48G 0 100% /.

This might have been caused by you being in the wrong directory when you un-tarred the archive, or it might be something else that's consuming all of this.

To find out what it is, start in the root filesystem cd / and run (as root) du -sm * to see the usage of each subdirectory in MB. Then cd into the one that's using the most and run the same command again, and keep drilling down until you identify the culprit.

Although you will likely want to ignore /home as those multi-TB disks are mounted on there. So just for the root filesystem the comamnd will have to be something like for i in $(ls -1 | grep -v ^home); do du -sm $i; done

profile picture
EXPERT
Steve_M
answered 5 months ago
profile picture
EXPERT
MaxCloud
reviewed 5 months ago
  • tu
    5 months ago

    Sorry, here is the output for df -h again:

    Filesystem      Size  Used Avail Use% Mounted on
udev             79G   12K   79G   1% /dev
tmpfs            16G  404K   16G   1% /run
/dev/xvda1       50G   48G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none             79G     0   79G   0% /run/shm
none            100M     0  100M   0% /run/user
overflow        1.0M     0  1.0M   0% /tmp
/dev/xvdg       8.8T  7.4T  990G  89% /home/ubuntu/users
/dev/xvdh       3.9T  1.9T  1.9T  50% /home/ubuntu/proc

    I tried checking root files system but don't really see an outlier or a significantly bigger file folder there... not sure what I'm looking for.

    At the mean time, I asked people from my group to try cleaning up and deleting/ transferring files to S3 where possible and one person replied "I tried to clean it, but the remaining space doesn’t increase"?!

  • Steve_M EXPERT
    5 months ago

    What's the output of running as root cd /; for i in $(ls -1 | grep -v ^home); do du -sm $i; done ?

  • tu
    5 months ago
    12	bin
68	boot
1	dev
10	etc
0	initrd.img
0	initrd.img.old
267	lib
1	lib64
1	lost+found
1	media
1	mnt
282	opt
du: cannot access ‘proc/121349/task/121349/fd/4’: No such file or directory
du: cannot access ‘proc/121349/task/121349/fdinfo/4’: No such file or directory
du: cannot access ‘proc/121349/fd/4’: No such file or directory
du: cannot access ‘proc/121349/fdinfo/4’: No such file or directory
0	proc
1	root
1	run
12	sbin
1	scratch
1	srv
0	sys
0	tmp
8676	usr
6230	var
0	vmlinuz
0	vmlinuz.old
  • Steve_M EXPERT
    5 months ago

    So this means its likely something in /home that is filling up the root filesystem,so need to try to find out what that is without traversing the two multi-TB filesystems.

    Can you temporarily unmount these? As root umount /home/ubuntu/users ; umount /home/ubuntu/proc so that what's left under /home is part of the root filesystem, and examine this as root cd /home ; du -sm *

    If those other filesystems cant be unmounted then examine /home while excluding them. As root cd /home ; for i in $(ls -1 | grep -v ^ubuntu); do du -sm $i; done

    If you have no luck there then as root cd /home/ubuntu ; for i in $(ls -1 | egrep -v ^users|^proc); do du -sm $i; done

  • tu
    5 months ago

    i'm able to unmount /home/ubuntu/proc but not /home/ubuntu/users. When examine /home I get:

    1280	linuxbrew
1	sgeadmin
5057904	ubuntu

    and going to ubuntu, I get:

    2829	anaconda3
6837	anaconda3.5
121	aws
1	bin
1	downloads
1	ebs
9	edirect
1	jupyter
1	kys
15905	miniconda3
3	pear
1	proc
1	reference_data
1	tmp
1	tools
5026222	users
1	xy43storefolder

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content