Disk space issue for build executor in AWS Blu Age Devops Pipeline
Hello, we have an issue on Jenkins CI with Build Executors :
Test case are not running due to some disk space issue, displaying an offline node, with the message of low disk space :
Moreover there is a console.log of a test case which results in a No space left.
So the question is, is there supposed to be a auto clean-up thing that is not working in our case, or how could we process the cleaning up in order to make our CI operational ?
Thank you for your response !
- Language
- English
asked 2 years ago209 views
- Newest
- Most votes
- Most comments
Hello,
Depending on the number of test cases you have in your project, the Jenkins EC2 instance may grow in size quickly, which often happens when there are many test cases generating reports. These reports can take up a lot of space over time. From what you've mentioned, it seems like the ec2-<AWS_REGION>-jenkins instance is running out of disk space. You can fix this by increasing the size of the root volume. Here's how:
- Go to your EC2 instance details and navigate to the Storage tab.
- Click on your Volume ID
- Select your volume, then click on Actions > Modify Volume
- Modify the size of your volume and save the changes
You may need to restart the EC2 instance, though in most cases, a reboot isn’t necessary.
Hope this helps! 😊
The issue you're experiencing with your Jenkins CI build executors running out of disk space is a common problem in CI/CD pipelines, especially when dealing with large builds or frequent executions. There are a few potential solutions to address this:
-
Increase disk space: If possible, you could increase the disk space allocated to your build environment. This might involve using a larger compute type with more available disk space for your build containers.
-
Implement regular cleanup: While there isn't typically an automatic cleanup process built into Jenkins or AWS CodeBuild (which is often used with AWS Blu Age pipelines), you can implement your own cleanup steps. Add a post-build step or a separate job that regularly cleans up old build artifacts, logs, and temporary files.
-
Optimize your build process: Review your build scripts and test cases to see if there are any unnecessary files being created or if large temporary files can be removed during the build process.
-
Use ephemeral build agents: Consider using ephemeral build agents that are created fresh for each build and destroyed afterward. This ensures you start with a clean slate each time.
-
Implement disk space checks: Add a pre-build step that checks available disk space and fails the build early if there isn't enough space, preventing partial builds that consume resources without completing.
-
Utilize AWS Systems Manager: If you're using AWS, you could leverage the AWSPremiumSupport-TroubleshootEC2DiskUsage runbook to analyze your disk usage and potentially automate the extension of partitions and file systems.
To make your CI operational again, start by implementing a cleanup job that removes old build artifacts and logs. Then, monitor your disk usage closely and consider implementing some of the longer-term solutions mentioned above to prevent this issue from recurring.
Remember to test any changes in a non-production environment first to ensure they don't negatively impact your build processes.
Sources
Troubleshooting AWS CodeBuild - AWS CodeBuild
Using SAW to diagnose common issues in your AWS environment | AWS re:Post
Relevant content
asked 2 years ago
asked 2 years ago
