How can I efficiently migrate a large MongoDB collection (with documents exceeding 16MB) to Amazon S3 using AWS DMS without failures?

Below are some pratice you can take as reference:

1. Handling Large BSON Documents • Split Large Documents: If possible, consider splitting large MongoDB documents into smaller sub-documents before migration. This can be done programmatically using MongoDB's aggregation framework or custom scripts. • Use LOB Mode: AWS DMS supports migrating large objects (LOBs) by enabling LOB mode. You can adjust the LobChunkSize and MaxLobSize parameters to handle large BSON documents more effectively. • Enable Compression: Compressing the data before migration can help reduce the size of BSON documents.
2. Adjusting DMS Settings • NestingLevel: If your documents are deeply nested, consider using "NestingLevel": "one" instead of "none". This can help flatten the structure and reduce size issues. • Task Settings: Fine-tune the CdcMinFileSize and CdcMaxBatchInterval values to optimize the CDC process for large documents. • Output Format: Switching to JSON instead of Parquet might simplify the migration process, as JSON is more flexible with document structures.
3. Alternative AWS Services • AWS DataSync: For large datasets, AWS DataSync can be a better alternative. It supports high-speed data transfers and can handle large files efficiently. • MongoDB Atlas Data Federation: If you're using MongoDB Atlas, you can leverage its Data Federation feature to export data directly to S3 in Parquet format. • AWS Snowball: For extremely large datasets, AWS Snowball provides a physical device for secure data transfer to S3.
4. Segmentation for Performance • Use segmentation to improve performance during migration. AWS DMS supports auto-segmentation and range segmentation for MongoDB collections, which can help distribute the load and avoid bottlenecks.
5. Error Handling and Monitoring • Implement robust error handling to retry failed migrations. • Use AWS CloudWatch to monitor DMS tasks and identify potential issues early.