By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

How to setup ongoing replication from Redshift to Redshift Serverless?

0

We are migrating from Redshift to Redshift Serverless to lower costs. I would like to have ongoing replication from RS to RS Serverless so that they are in sync and we can do blue/green rollout by gradually increasing traffic to RS Serverless, validate it then do the cutoff.

For the blue/green rollout we will be using ALB weighted routing, which is connected to 2 target groups: 1 target group is a Lambda that queries Redshift, the other target group is another Lambda that queries Redshift Serverless.

I am planning to use AWS DMS for the replication, but it seems that supported Sources don't include Redshift: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Sources.html

Redshift and Redshift Serverless are only supported as Targets: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html

Snapshotting Redshift and restoring to Redshift Serverless will probably NOT include the last minutes/hours worth of data, which is not ideal for me. I would like all data to be in sync, so that when I do the cutoff, all data is there.

**How can I create ongoing replication between Redshift and Redshift Serverless? **

I would like to avoid writing code for this, though I know it can be done that way, but it would be my last option.

Topics
Migration & ModernizationServerlessComputeDatabaseAnalytics
Tags
AWS Database Migration ServiceAWS LambdaDatabaseAmazon RedshiftAmazon Redshift Serverless
Language
English
Denis Rumnici
asked a month ago513 views
2 Answers
1
Accepted Answer

The easiest option is to enable data sharing between provisioned and serverless instance.

  1. Take a snapshot of provisioned cluster and restore to serverless. This will be your baseline.
  2. Setup data sharing between provisioned and serverless instance.
  3. Run "insert into" scripts to copy data from data share tables to local tables in serverless. Hope you have audit fields to pull only incremental records. If some tables do not, then a full table refresh is required for such tables.

Also for Blue/Green scenario, at the time of cut-over, please consider following steps.

  • Please disconnect all inbound and outbound connnections to old instance and route to serverless instance.
  • Pause the provisioned cluster (You can query data from consumer serverless instance even though producer is paused)
  • Ensure you copied all the data from datashare into your local tables on serverless.
  • Take final snapshot for any future reference and then delete the old provisioned instance.
AWS
EXPERT
Satesh_Sonti-Redshift-DataZone
answered a month ago
profile picture
EXPERT
Giovanni Lauria
reviewed a month ago
  • Denis Rumnici
    a month ago

    "Run "insert into" scripts to copy data from data share tables to local tables in serverless."

    How do I do that? Is it configured from the data share or I have to write some custom Lambda for it?

  • Denis Rumnici
    a month ago

    I accepted the answers from the 2 comments of Satesh Sonti-Redshift.

    It seems it would work, but I decided that we should go with a different solution, that is specific to our team: Currently we update Redshift data through "COPY" statements that copy data from an S3 bucket into Redshift. This statement is executed in one of our Lambda functions. We will change this Lambda function code to send updates not only to Redshift but also to the new Redshift Serverless that we will create (through the CDK). To do this we will use the RedshiftDataClient class.

    This way all new data is uploaded to both RS and RS Serverless. They are in sync.

    Thank you for the support Satesh, highly appreciated.

0

@Denis, You can run the insert into statements from stored procedure or lambda. Either way is fine. If it is one time, you can just run them from query editor. I would suggest stored procedure and schedule with Query editor V2 for running it repeatedly.

Reference: https://docs.aws.amazon.com/redshift/latest/mgmt/query-editor-v2-schedule-query.html https://docs.aws.amazon.com/redshift/latest/dg/performing-a-deep-copy.html

AWS
EXPERT
Satesh_Sonti-Redshift-DataZone
answered a month ago
profile picture
EXPERT
Adeleke Adebowale J
reviewed 9 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content