Can I load data to redshift serverless directly fom SQS

0

I have some json files being loaded to s3 and the events are being queued to sqs, I need to copy the information to redshift, I want to know if there is a way to load it directly from the sqs or if I must have a lambda or glue service running in the middle to retrieve the sqs.

2 Answers
0

You're correct - you need some "middle" service that does the data ingest for you. Glue and Lambda (in that order) are probably the right things to look at but there are other ways such as running the data import in a container or on an EC2 instance.

profile pictureAWS
EXPERT
answered a month ago
0

You definitely have several optiions:

  • You can use AWS Lambda to retrieve messages from the SQS queue. The Lambda function would then load the data into Redshift using the COPY command.
  • You can use AWS Glue to orchestrate the data flow. A Glue trigger would be configured to watch the SQS queue. When a message is received, Glue would retrieve it and use a Glue job to load the data into Redshift.
  • The AWS Data Pipeline service can also be used to set up a data flow that reads from SQS and loads into Redshift. A pipeline would be created with an SQS source and a Redshift destination.
  • In all the above cases, the data would need to be first written to an S3 bucket from the Lambda/Glue job before using the COPYcommand to load it into Redshift. Redshift only supports loading data directly from S3 but not SQS. So an intermediate processing step is required to retrieve the messages from SQS and stage the data in S3.
profile picture
EXPERT
answered a month ago
AWS
SUPPORT ENGINEER
reviewed 10 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions