CDK S3 Bucket growing out of control since using 'cdk gc --unstable=gc' - garbage collection

Issue Analysis:

• The CDK asset bucket is used to store temporary deployment assets (e.g., Lambda package files, CloudFormation templates).
• Normally, older assets should be automatically deleted, but running cdk gc --unstable=gc appears to have caused a sharp increase in stored objects.
• Lifecycle rules on the bucket were unchanged (default: deleting objects older than 30 days).
• The issue persisted across multiple deployments, with object count and bucket size continuing to grow abnormally.

Potential Causes:

• • CDK Garbage Collection Behavior:

• The cdk gc command may not be functioning as expected and could be leaving behind additional metadata or duplicate assets.

• Asset Retention Settings in CDK:

• If assets are not correctly marked for deletion, they may remain indefinitely.

• • Deployment Frequency Impact:

• Since multiple deployments occur per day, new assets keep accumulating, exacerbating the issue.

• Stale Assets Not Getting Cleaned Up:

• CDK might not be correctly tracking which assets are in use, leading to excessive storage.

Solution & Workarounds:

1. Manually Clean Up the Bucket (Safe Approach)

• Before deleting anything, verify that the bucket contains only deployment artifacts and not any critical assets.
• Use an S3 lifecycle policy to automatically delete objects after a shorter duration (e.g., 7 days instead of 30 days). Example policy:

{
    "Rules": [
        {
            "ID": "AutoDeleteOldCDKAssets",
            "Prefix": "",
            "Status": "Enabled",
            "Expiration": {
                "Days": 7
            }
        }
    ]
}

• If immediate cleanup is needed, manually delete old assets via AWS CLI:

aws s3 rm s3://your-cdk-bucket-name --recursive

Be careful not to delete assets currently in use.

2. Adjust CDK Asset Retention Policy

Ensure that cdk destroy properly cleans up resources. Run the following command to check retained stacks:

cdk list

Destroy unused stacks:

cdk destroy stack-name

3. Investigate and Use Stable CDK Commands

Instead of cdk gc --unstable=gc, use:

cdk synth && cdk deploy

If garbage collection is required, test with the latest stable CDK version.

4. Upgrade CDK to the Latest Version

This issue might be a bug in a specific CDK version. Try upgrading:

npm install -g aws-cdk

Verify the installed version:

cdk --version

5. Monitor Bucket Growth Over Time

• Use CloudWatch metrics to track bucket size (BucketSizeBytes) and object count (NumberOfObjects).
• Set up an alert for unexpected growth trends.

Final Thoughts

• It is generally safe to delete unnecessary objects from the CDK asset bucket if you are sure they are not in use.
• Enabling a strict lifecycle policy for automatic cleanup is recommended.
• AWS CDK's gc behavior should be used cautiously, as it may not reliably remove old assets.