Skip to content

Importing DynamoDB data from one organisation account to another organisation account

0

We currently have DynamoDB tables on an organisation account (account 1) that we have now managed to control via CDK. We then created a new organisation account (account 2) with DynamoDB and used the CDK to deploy the tables to that account. I have since turned on PITR on account 1 and done a cross account export to S3 to a bucket on account 2. SO i now have my DynamoDB data from account 1 sitting in an S3 bucket on account 2. I want to import this data to the relevant DynamoDB tables in account 2, however, it seems that from the Import from S3, it won't allow me to import the data into an existing table, it will create a new table as part of the import. This would mean creating tables that are not controlled by CDK.

Is there a way to import DynamoDB data that is stored in an S3 bucket into the relevant DynamoDB table?

asked a year ago344 views

2 Answers
1

You're correct that the DynamoDB Import from S3 feature has a limitation - it only creates new tables and doesn't support importing into existing tables. This is indeed a challenge when you want to maintain your CDK-managed infrastructure.

For your scenario where you need to import data into existing CDK-managed tables in account 2, you have a few options:

  1. AWS Data Pipeline approach: You can create a pipeline in account 2 to import the data from your S3 bucket into your existing DynamoDB tables. This was the traditional method before newer features were introduced. This approach allows you to import into existing tables but will consume provisioned throughput on your target tables.

  2. Temporary solution with table replacement: You could:

  • Import the data from S3 to create new tables in account 2
  • Update your CDK code to reference these imported tables instead of creating new ones
  • Once your CDK recognizes the existing tables, you can continue managing them through CDK
  1. Custom ETL process: Develop a custom solution using AWS services like Lambda or AWS Glue to read the data from S3 and write it to your existing DynamoDB tables. This gives you more control over the process but requires additional development.

Each approach has trade-offs in terms of development effort, cost, and impact on your infrastructure-as-code practices. The best approach depends on factors like the size of your data, your timeline, and your team's expertise.

If maintaining CDK control is critical, you might want to consider the custom ETL approach, as it allows you to keep your existing table structure while still importing the data from S3.
Sources
Using DynamoDB with Amazon S3 to export and import table data - AWS Prescriptive Guidance
Migrating a DynamoDB table from one account to another - Amazon DynamoDB
Using AWS Data Pipeline - AWS Prescriptive Guidance

answered a year ago

EXPERT

reviewed a year ago

1

In CDK you must create the table while pointing it to the S3 Source using importSource: https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_dynamodb.Table.html#importsource

The alternative approach build the tables with S3 data manually, then import them to the stack using cdk import: https://docs.aws.amazon.com/cdk/v2/guide/ref-cli-cmd-import.html

AWS
EXPERT

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.