Skip to content

DMS Migration from conditional writes is slow from SQL to Dynamodb

0

Hi Team,

I am migrating the database from SQL to Dynamodb using the conditional writes ,But its slow. Without the conditional writes its fast on migration ...I have tried to tune the parameters of ParallelLoadThreads & ParallelLoadBufferSize ,No seeing much performnace improvement in the DMS Task.

Regards, Balaji T P

asked a year ago174 views

1 Answer
0

When migrating from SQL to DynamoDB using AWS DMS, conditional writes can indeed slow down the migration process compared to standard writes. This is expected behavior since conditional writes require additional validation checks before data is committed.

To improve performance for your DMS migration to DynamoDB, consider these approaches:

  1. Parallel load optimization:
  • You've already tried adjusting ParallelLoadThreads (which can be set up to 200, with support able to increase this limit if needed)
  • You've also adjusted ParallelLoadBufferSize (which can be set up to 1,000)
  • Note that these settings only help with loading data to the target table using multiple threads but don't help unload the source data in parallel
  1. For large tables, consider these additional optimizations:
  • Test with a replica of your production instance to establish a baseline for full load time
  • Consider parallelizing the full load by using multiple tasks (since parallel-load table level mapping isn't supported for DynamoDB targets)
  1. For CDC (Change Data Capture) performance, if you're using it:
  • Adjust batch processing settings if BatchApplyEnabled is true
  • Consider setting BatchApplyPreserveTransaction to false if you can afford temporary lapses in transactional integrity for better performance
  • Adjust BatchApplyMemoryLimit (default is 500MB) for pre-processing in batch optimized apply mode
  1. Validation settings:
  • If you're using validation, consider disabling it during the initial migration and running a separate validation-only task afterward
  • For validation-only tasks, you can increase PartitionSize and ThreadCount to speed up validation

Since conditional writes inherently add overhead to the process, you might need to accept some performance trade-offs if these conditional checks are essential for your data integrity requirements.
Sources
Step-by-step Amazon RDS for MySQL database to Amazon DynamoDB migration walkthrough - Database Migration Guide
Change processing tuning settings - AWS Database Migration Service
Data masking and performance improvements in AWS DMS 3.5.4 | AWS Database Blog

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.