Transfer from S3 to Redshift and filter specific key

Question

Hey!
My first task with AWS, so the question is more about the best way to solve the problem.
The problem:
   I have integration from some service to export data to S3. After each export I need to transfer the data to Redshift and to filter some specific keys.

I checked those ways to do it:

1. Lambda function to filter out the key
2. Batch 
3. Transfer with airflow and filter the key by the list in the manifest

I’m checking now the 3rd option.

What do you think?

Answer

Hi, 
have you considered using Redshift Spectrum to import data into Redshift using SQL?

Assuming the data is in the format that can be read by Redshift Spectrum it would greatly simplify your ingestion pipeline.

You can schedule the query that does the import to automate the process https://docs.aws.amazon.com/redshift/latest/mgmt/query-editor-schedule-query.html

Answer

You may want to consider AWS Glue, which is a serverless data integration service.

Load data from Amazon S3 to Amazon Redshift using AWS Glue - https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/load-data-from-amazon-s3-to-amazon-redshift-using-aws-glue.html

Transfer from S3 to Redshift and filter specific key

Relevant content