Issue with Increasing Data Volume in DocumentDB 5.0

The issue with the VolumeBytesUsed metric doubling could be due to various factors even with DocumentDB 5.0.0. Here are some steps and considerations to help troubleshoot and potentially resolve the issue:

1. Check DocumentDB Storage Optimization Ensure that your DocumentDB instance is optimizing storage correctly. DocumentDB 5.0.0 should handle dynamic resizing of storage, but there may be specific configurations or operations that impact this.

2. Monitor Data Retention and Deletion Processes Even though you delete data older than 6 months daily, ensure that the deletions are actually reducing the storage usage. Sometimes, deleted data might not be immediately removed from the storage due to how DocumentDB manages its internal storage.

3. Analyze Snapshots and Backup Frequency of Snapshots: Daily snapshots and retaining them for 24 hours should not typically cause a significant increase in storage usage. However, verify if the snapshots are being correctly managed and deleted after the retention period. Snapshot Size: Check the size of the snapshots. If they are unexpectedly large, it could indicate underlying issues with data storage. 4. Indexes and Collections Indexes: Ensure that indexes are optimized. Unused or redundant indexes can increase storage usage significantly. Collections: Review the collections and their respective storage sizes. Look for any unexpected increases in size that might indicate issues such as data not being deleted properly. 5. Performance Insights Use performance insights and logs to identify any unusual operations or patterns that might be contributing to increased storage usage. Look for:

Large batch operations Frequent updates or upserts Large document sizes 6. AWS Support If the issue persists, it might be best to contact AWS Support directly. Provide them with detailed logs and metrics. They can offer more specific insights and solutions based on the internal workings of DocumentDB.

Additional Steps Here are some detailed steps to help you gather more information and possibly resolve the issue: ** Review Metrics and Logs:**

Use Amazon CloudWatch to monitor VolumeBytesUsed and other related metrics. Check DocumentDB logs for any warnings or errors related to storage. Evaluate Storage Metrics:

Check the DatabaseFreeStorageSpace metric to understand how much free storage is available. Compare VolumeBytesUsed over time to identify any sudden spikes or patterns. Run Storage Cleanup Commands:

Consider running the compact command on collections to reclaim storage space. Ensure that data deletion scripts are running correctly and verify their impact on storage usage. Review Storage Settings:

Check the storage settings in the DocumentDB cluster configuration. Ensure that the automatic scaling settings are correctly configured.

import boto3
import datetime

# Initialize a session using Amazon DocumentDB
client = boto3.client('cloudwatch')

# Specify the DocumentDB cluster identifier
cluster_identifier = 'your-cluster-identifier'

# Specify the start and end time for the metrics
end_time = datetime.datetime.now()
start_time = end_time - datetime.timedelta(days=90)

# Get the VolumeBytesUsed metric
response = client.get_metric_statistics(
    Namespace='AWS/DocDB',
    MetricName='VolumeBytesUsed',
    Dimensions=[
        {
            'Name': 'DBClusterIdentifier',
            'Value': cluster_identifier
        },
    ],
    StartTime=start_time,
    EndTime=end_time,
    Period=86400,  # Daily data points
    Statistics=['Average'],
)

# Process the response
data_points = response['Datapoints']
data_points.sort(key=lambda x: x['Timestamp'])

for dp in data_points:
    print(f"Date: {dp['Timestamp']}, VolumeBytesUsed: {dp['Average']}")

This script fetches and prints the VolumeBytesUsed metric for the past 90 days. You can use this data to visualize trends and identify any sudden increases in storage usage.

By following these steps and considerations, you should be able to gather more information about the storage usage issue in your DocumentDB instance and take appropriate actions to resolve it.