how to back up data from dynamo db to s3 with point in time recovery enabled?

Question

I tested following code in a smaller table , which works, but when i tried same command with table that has 50k and higher , it seems it starts the process but i only see files saved to s3 by this process which are only few hundred KB. I waited and see, if it takes longer , since i have 50 thousands records , but no luck.

is there any limit on the size of the dynamodb, for following command to succeed?

also, under dynamdb -> export to s3 page in my aws account, all the entries show as completed, but i don't see the data being saved successfully in the s3 bucket path , i specified.

```
aws dynamodb export-table-to-point-in-time --table-arn "...." --s3-bucket "...." 
```

re:Post Agent · Answer

To back up data from DynamoDB to S3 with point-in-time recovery (PITR) enabled, you need to follow these steps:

1. First, ensure PITR is enabled on your DynamoDB table. This is a prerequisite for using the export to S3 feature. If PITR isn't enabled, your export request will fail with a "PointInTimeRecoveryUnavailableException" error.

2. Set up proper IAM permissions. Your IAM policy needs to allow both the DynamoDB export action (`dynamodb:ExportTableToPointInTime`) and S3 actions (`s3:AbortMultipartUpload`, `s3:PutObject`, and `s3:PutObjectAcl`).

3. Use the export command you've been using:
```
aws dynamodb export-table-to-point-in-time --table-arn "..." --s3-bucket "..."
```

Regarding your issues with larger tables:

There are no documented size limits for DynamoDB exports to S3. The export feature is designed to handle tables with hundreds of terabytes of data. The exports are asynchronous and don't impact table performance.

If your exports show as "completed" in the AWS console but you're not seeing the expected data in S3, check these potential issues:

- Verify you have the correct permissions to both write to the S3 bucket and view its contents
- Check if you're looking in the correct S3 path/prefix where the export was directed
- Confirm that the export process actually completed successfully by checking for any error messages in the export details
- Remember that exports are in either DynamoDB JSON or Amazon Ion format, so the files might not be directly readable without proper parsing

The export process is asynchronous, so for larger tables, it may take some time to complete. When the process shows as "completed" in the console, the data should be available in the specified S3 bucket.  
**Sources**  
[DynamoDB data export to Amazon S3: how it works - Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport.HowItWorks.html)  
[Amazon DynamoDB point-in-time recovery (PITR)](https://aws.amazon.com/dynamodb/pitr/)  
[Requesting a table export in DynamoDB - Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataExport_Requesting.html)

Shahed Kazi (@shajam) · Answer

I have run this command successfully on multiple TBs DynamoDB tables. Note, it takes time ~30 mins. 
If you are running this repeatedly, then run this using Step Function and Lambda, will be easier to manage and schedule.

how to back up data from dynamo db to s3 with point in time recovery enabled?

新增您的答案

相關內容