How to upload a parquet format file of RDS table data to S3 without using snapshots

Question

I'm trying to send a copy of an AWS RDS table data to an external cloud storage service to do some analysis. (The reason I'm using the storage service is because I want to use it as a data lake.) 
I prefer Parquet to CSV as the file format of the copied data . （Parquet is preferred over CSV because it is easier to transform.) I think snapshot of RDS is the best solution, but snapshot can't specify table columns, and it outputs columns that should be masked as they are, so there is a security concern.

So, I would like to know if there is a way to output RDS table data to S3 as a Parquet file without using snapshot. Thank you very much for your help.

Answer

In addition to the Save to S3 option, you can also write a Glue job to query RDS data and store in S3 in Parquet format. Some helpful information [here](https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/three-aws-glue-etl-job-types-for-converting-data-to-apache-parquet.html).

Answer

Please take a look at this https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Integrating.SaveIntoS3.html,
You can selectively send data to an s3 file, However this is text/csv file, 
You can have a lambda that will trigger when a file is generated and that will convert your text/csv file to parquet format.

How to upload a parquet format file of RDS table data to S3 without using snapshots

相關內容