- Newest
- Most votes
- Most comments
When migrating from SQL to DynamoDB using AWS DMS, conditional writes can indeed slow down the migration process compared to standard writes. This is expected behavior since conditional writes require additional validation checks before data is committed.
To improve performance for your DMS migration to DynamoDB, consider these approaches:
- Parallel load optimization:
- You've already tried adjusting
ParallelLoadThreads(which can be set up to 200, with support able to increase this limit if needed) - You've also adjusted
ParallelLoadBufferSize(which can be set up to 1,000) - Note that these settings only help with loading data to the target table using multiple threads but don't help unload the source data in parallel
- For large tables, consider these additional optimizations:
- Test with a replica of your production instance to establish a baseline for full load time
- Consider parallelizing the full load by using multiple tasks (since parallel-load table level mapping isn't supported for DynamoDB targets)
- For CDC (Change Data Capture) performance, if you're using it:
- Adjust batch processing settings if BatchApplyEnabled is true
- Consider setting BatchApplyPreserveTransaction to false if you can afford temporary lapses in transactional integrity for better performance
- Adjust BatchApplyMemoryLimit (default is 500MB) for pre-processing in batch optimized apply mode
- Validation settings:
- If you're using validation, consider disabling it during the initial migration and running a separate validation-only task afterward
- For validation-only tasks, you can increase PartitionSize and ThreadCount to speed up validation
Since conditional writes inherently add overhead to the process, you might need to accept some performance trade-offs if these conditional checks are essential for your data integrity requirements.
Sources
Step-by-step Amazon RDS for MySQL database to Amazon DynamoDB migration walkthrough - Database Migration Guide
Change processing tuning settings - AWS Database Migration Service
Data masking and performance improvements in AWS DMS 3.5.4 | AWS Database Blog
answered a year ago
Relevant content
asked 3 years ago
