Inquiry about Automating Aurora PostgreSQL Snapshot Export to S3 with AWS DMS

Question

Dear Everyone,
I am exploring options for automatically exporting point-in-time data from my Amazon Aurora PostgreSQL database to an S3 bucket. I am currently using AWS Backup for creating regular snapshots of my Aurora PostgreSQL cluster.
I would like to understand if AWS Database Migration Service (DMS) can be directly used to automate the export of these RDS snapshots (created by AWS Backup) to an S3 bucket.
My understanding is that DMS primarily works with live database instances as sources. However, I have come across discussions that suggest potential workarounds or configurations might exist to achieve a similar outcome using DMS.
Could you please clarify the following:
1.	Can AWS DMS directly connect to or utilize an RDS snapshot (not a restored instance) as a source for a migration task to an S3 target endpoint?
2.	If the answer to the above is no, are there any recommended or supported methods using DMS to automate the export of data representing a point-in-time snapshot of my Aurora PostgreSQL database to S3, without first restoring the snapshot to a temporary RDS instance?
3.	If using DMS requires restoring a snapshot to a temporary instance, what are the best practices and considerations for automating this entire workflow (restoration, DMS migration to S3, and cleanup of the temporary instance)?
4.	Are there any specific DMS configurations, endpoint settings, or task settings that are particularly relevant for exporting data to S3 in a way that represents a database snapshot?
5.	Any recommend for use anther service or anther way for automate export an RDS aurora PostgreSQL snapshot to S3 
My goal is to have a cost-effective and automated way to archive or process historical data from my Aurora PostgreSQL database in S3, ideally reflecting the state of the database at the time of the snapshot.
Thank you for your time and assistance with this inquiry.
Sincerely,

re:Post Agent · Answer

Thank you for your inquiry about automating Aurora PostgreSQL snapshot exports to S3 using AWS DMS. I'll address your questions and provide some recommendations:

1. AWS DMS cannot directly use an RDS snapshot as a source for migration tasks to an S3 target endpoint. DMS is designed to work with live database instances as sources.

2. There is no direct method using DMS to automate the export of data representing a point-in-time snapshot of your Aurora PostgreSQL database to S3 without first restoring the snapshot to a temporary RDS instance.

3. If you choose to use DMS by restoring a snapshot to a temporary instance, you would need to create an automated workflow that includes:
   - Restoring the snapshot to a temporary RDS instance
   - Configuring and running a DMS task to migrate data from the temporary instance to S3
   - Cleaning up the temporary instance after the migration is complete
   This process can be automated using AWS Lambda functions triggered by CloudWatch Events or Step Functions to orchestrate the workflow.

4. When configuring DMS for exporting data to S3, consider the following settings:
   - Use the "S3" target endpoint type
   - Configure appropriate IAM roles for DMS to access your S3 bucket
   - Set up task settings to control how data is formatted and partitioned in S3
   - Use CDC (Change Data Capture) if you need to capture ongoing changes

5. For a more direct and potentially cost-effective approach to exporting Aurora PostgreSQL snapshots to S3, consider using the native snapshot export feature provided by Amazon RDS:
   - This feature allows you to export DB snapshots directly to Amazon S3
   - The export process runs in the background and doesn't affect database performance
   - Exported data is stored in Apache Parquet format, which is efficient for analytics
   - You can export all types of DB snapshots, including manual, automated, and AWS Backup snapshots
   - To automate this process, you can use AWS CLI commands or AWS SDKs in combination with Lambda functions and CloudWatch Events

To set up an automated snapshot export to S3:
1. Create an IAM role that grants the snapshot export task access to your S3 bucket
2. Create a symmetric encryption AWS KMS key for server-side encryption
3. Use the AWS CLI `start-export-task` command or the equivalent API call in your automation script
4. Monitor the export task progress and handle any failures in your automation workflow

This native snapshot export feature aligns well with your goal of having a cost-effective and automated way to archive historical data from your Aurora PostgreSQL database in S3, reflecting the state of the database at the time of the snapshot.

Remember to consider data retention policies, encryption requirements, and access controls when implementing this solution.
      
**Sources**  
[Exporting DB snapshot data to Amazon S3 for Amazon RDS - Amazon Relational Database Service](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ExportSnapshot.html)  
[Exporting DB cluster snapshot data to Amazon S3 - Amazon Aurora](https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-export-snapshot.html)  
[Amazon RDS examples using AWS CLI - AWS SDK Code Examples](https://docs.aws.amazon.com/code-library/latest/ug/cli_2_rds_code_examples.html)

Inquiry about Automating Aurora PostgreSQL Snapshot Export to S3 with AWS DMS

Add your answer

Relevant content