Skip to content

Migration from GCP sql (postgres17.7) to AWS rds (postgres17.7) using AWS dms

0

the source database is publicly accessible , i connected via DMS replication instance and started migration , where i got some error on table , so. i excluded that and included some table which are not failed ; i choose full mode and the LObchunksize was 64kb, and Maximum LOB size (KB) 0. ( i choose full mode because column size i have are greater, i dont know exactly the maximum column size i have, and to escape from truncation issue, i going with full mode) i ran the migration task metrics on DMS Enhanced monitoring

my source db is public , so i use its ip as endpoint and in rds i given global certificate to authenticate with in endpoints .

i have taken schema only dump manually from ec2 and restored in RDs postgres

Table Size on SourceMigrated Data so far
8515 MB164 MB
7697 MB102 MB
1554 MB64 MB

Running: its been runing 18 hours , this was the data so far migrated

in source GCPsql (postgres 17):

  1. i haven't added any parameter in target rds , but in source GCP i did add "cloudsql.logical_decoding and cloudsql.enable_pglogical"

in target RDS (postgres17):

  1. i haven't added any parameter so far

looking for minimal downtime, and right parameters to pass in parameter groups in source and target

2 Answers
0

To improve your PostgreSQL migration from Google Cloud SQL to Amazon RDS using AWS DMS, you need to optimize several parameters and consider a few key adjustments.

For your current "migrate only" (full load) approach:

  1. The slow migration speed you're experiencing (only a small fraction of data transferred after 18 hours) suggests performance issues. For large tables like yours (8515 MB, 7697 MB, etc.), consider:
  • Dividing large tables into separate tasks using selection rules
  • Monitoring disk and CPU usage on the target side
  1. For LOB handling, your settings (LOBChunkSize of 64KB and Maximum LOB size of 0) are appropriate for full mode if you have large objects, as this allows unlimited LOB sizes.

For when you move to "migrate and replicate" (full load + CDC):

Source database (Google Cloud SQL) parameters:

  • You've already enabled "cloudsql.logical_decoding" and "cloudsql.enable_pglogical" which is good
  • Ensure binary logging is properly configured

Target database (Amazon RDS) parameters:

  • Increase max_logical_replication_workers
  • Increase max_sync_workers_per_subscription
  • These parameters help speed up initial data syncing during replication

Additional considerations:

  1. Database extensions are not replicated as part of pglogical replication. You'll need to manually create them in your target RDS instance.
  2. For minimal downtime, after your full load completes, you can set up CDC to capture ongoing changes while your application still uses the source database.
  3. When using CDC, you'll need to provide a native start point or choose "Immediately" in the data migration settings.
  4. Monitor replication status and CDC latency before performing the final cutover.
  5. Remember that AWS DMS creates a serverless environment for the migration process, using pg_dump/pg_restore for schema migration and a publisher/subscriber model for ongoing replication.

For your specific case with large tables and public endpoints, ensure your security groups and network settings allow proper connectivity between the DMS replication instance and both databases.
Sources
Migrating data from PostgreSQL databases with homogeneous data migrations in AWS DMS - AWS Database Migration Service
Migrate Google Cloud SQL for PostgreSQL to Amazon RDS and Amazon Aurora using pglogical | AWS Database Blog
Migrate a self-managed MySQL database to Amazon Aurora MySQL using AWS DMS homogeneous data migrations | AWS Database Blog
Step-by-step PostgreSQL database to Amazon RDS migration walkthrough - Database Migration Guide

answered 10 months ago

  • "Dividing large tables into separate tasks using selection rules" ==> i already did selection rule, where i specified the each table separately

    though i chosen, still being slow. do i missing anything or should i consider any other thing to check ?

    i ran this migration by creating single task , where i selected each table on selection rule

0

Your current DMS task is running very slowly because it’s using Full LOB mode with Max LOB size = 0, which forces DMS to stream each large object row-by-row without batching. That’s safe for truncation but extremely inefficient for multi‑GB tables. For initial migration, switch to Limited LOB mode with a realistic Max LOB size (for example, 1–2 MB if your largest LOBs fit) so DMS can batch and parallelize reads. You can then identify any outliers with larger columns and handle them separately if needed. Also make sure your replication instance has enough vCPU and memory (use at least a `dms.r5. —Taz

answered 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.