DMS truncation when redshift as target

0

i have one table in aurora db that i want to migrate to redshift there is one column in that table which store JSONB object and there is one data whose size is around 50KB.

my question is when i perform migration from aurora to redshift via DMS then that JSON object data is getting truncated to 31.5KB even though i have setted the MaxLobSize to 63KB with Limited LOB mode in migration task.

2 Answers
0

That sounds like a bug. Have you captured anything in the logs that you can share with AWS Support via a support ticket?

Also, there might be other easier ways to get this data synced to Amazon Redshift. Have you tried zero-ETL , refer https://aws.amazon.com/about-aws/whats-new/2024/03/amazon-aurora-zero-etl-integration-redshift-data-filtering-cloudformation/ , towards the bottom it provides a blog link with an example use case too.

profile pictureAWS
answered a month ago
profile picture
EXPERT
reviewed a month ago
  • In logs it is showing the truncated to 64KB but our actual data is getting truncated even less than 64KB. but anyways we are using Zero-ETL now as this is new feature which came just couple of days ago in our region. but here as well am facing some other issue when i create ETL integration from RDS my aurora postgres cluster is not getting listed here in the select source.Zero-ETL-integration this is my aurora cluster RDS aurora postgres cluster

  • Aurora PostgreSQL zero-ETL is in preview so there are certain limitations, listed at https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/zero-etl.html#zero-etl.reqs-lims-apg , as long as these are satisfied then it should show work for you. Kindly double check these and if still doesn't work then you can open Support Ticket or reach out to your Account Team and someone will be able to assist you further.

0

You can create target table manually in Redshift with "Super" data type. This data type allows you to store up to 16MB of semi-structured JSON data. If you're using AWS Database Migration Service (DMS) to replicate data from a source database to an Amazon Redshift target, the Super data type is supported. You can define the target table column type as Super in the DMS table mapping, the data will be replicated accordingly. For additional details on super data type please refer https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html

AWS
EXPERT
Nita_S
answered 25 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions