Backup interrupt for no space

The error "No space left on device" you're encountering during your daily backup of a Virtual Machine can be misleading, as it may not always indicate a lack of overall storage space. Even after increasing the volume space, the issue persists, which suggests there might be other factors at play.

Here are a few potential causes and solutions to consider:

1. Root filesystem full: Check if the root filesystem is full, even if other partitions have space. You can use the command 'df -h' to view disk usage across all filesystems. If the root partition (usually mounted at '/') is at 100% capacity, this could cause the backup to fail.

2. Inode exhaustion: Sometimes, even with available disk space, you can run out of inodes. Use 'df -i' to check inode usage. If inodes are exhausted, you'll need to delete unnecessary files or increase the number of inodes.

3. Temporary directory full: The backup process might be using a temporary directory that's full. Check the space in /tmp or any other temporary directories used in your backup process.

4. Snapshot issues: If you're using snapshots as part of your backup process, ensure that there's enough space for new snapshots and that old ones are being properly deleted.

5. File system corruption: In rare cases, file system corruption can cause these errors. Running a file system check might help.

6. Backup destination: Ensure that the destination where you're storing the backup has sufficient space.

To troubleshoot, you can:

1. Run 'du -sm *' in the root directory to identify which subdirectories are using the most space.
2. Check and clean up any large log files or temporary files that might be filling up the disk.
3. Verify that your backup process is not accidentally writing to a partition with limited space.
4. Consider increasing the size of the root partition if it's consistently running out of space.

If the issue persists after these checks, you may need to review your backup process and storage configuration to ensure they're optimized for your needs.
Sources
"Cannot write: No space left on device" error even though there is storage available | AWS re:Post
AWS CodeBuild Failed with Error: no space left on device | AWS re:Post

Hi,

In addition to the Agent answer, please be aware that physical extension of disk storage is not immediately accessible in your file system. You must also extend the corresponding disk partition.

See https://www.msp360.com/resources/blog/linux-resize-partition/ or https://www.alibabacloud.com/help/en/ecs/user-guide/extend-the-partitions-and-file-systems-of-disks-on-a-linux-instance

It's same with WIndows: https://learn.microsoft.com/en-us/windows-server/storage/disk-management/extend-a-basic-volume

Best,

Didier

Hi. Sorry for my questions, simple to some :-) Below is the size of the VM, I had allocated 200Gb.

I saw that the size of the last valid backup was 57Gb. Please, help me to understand how solve this problem. Thanks

BELOW the partition.... It seems to me that everything is okay with the dimensions of partitions

Disk /dev/loop0: 25.74 MiB, 26992640 bytes, 52720 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop1: 26.32 MiB, 27602944 bytes, 53912 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop2: 55.66 MiB, 58363904 bytes, 113992 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop3: 55.36 MiB, 58052608 bytes, 113384 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop4: 63.97 MiB, 67080192 bytes, 131016 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop5: 63.7 MiB, 66789376 bytes, 130448 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop6: 73.88 MiB, 77463552 bytes, 151296 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop7: 73.87 MiB, 77459456 bytes, 151288 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes GPT PMBR size mismatch (314572799 != 419430399) will be corrected by write. The backup GPT table is not on the end of the device.

Disk /dev/nvme0n1: 200 GiB, 214748364800 bytes, 419430400 sectors Disk model: Amazon Elastic Block Store
Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: B3F8B8BC-6206-4B08-8022-448584C72D0D

Device Start End Sectors Size Type /dev/nvme0n1p1 227328 314572766 314345439 149.9G Linux filesystem /dev/nvme0n1p14 2048 10239 8192 4M BIOS boot /dev/nvme0n1p15 10240 227327 217088 106M EFI System

Partition table entries are not in disk order.

Disk /dev/loop8: 87.04 MiB, 91267072 bytes, 178256 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop9: 89.4 MiB, 93745152 bytes, 183096 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop10: 38.83 MiB, 40714240 bytes, 79520 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/loop11: 44.3 MiB, 46448640 bytes, 90720 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes ubuntu@enti:~$