Can I load data to redshift serverless directly fom SQS

0

I have some json files being loaded to s3 and the events are being queued to sqs, I need to copy the information to redshift, I want to know if there is a way to load it directly from the sqs or if I must have a lambda or glue service running in the middle to retrieve the sqs.

2 個答案
0

You're correct - you need some "middle" service that does the data ingest for you. Glue and Lambda (in that order) are probably the right things to look at but there are other ways such as running the data import in a container or on an EC2 instance.

profile pictureAWS
專家
已回答 2 個月前
0

You definitely have several optiions:

  • You can use AWS Lambda to retrieve messages from the SQS queue. The Lambda function would then load the data into Redshift using the COPY command.
  • You can use AWS Glue to orchestrate the data flow. A Glue trigger would be configured to watch the SQS queue. When a message is received, Glue would retrieve it and use a Glue job to load the data into Redshift.
  • The AWS Data Pipeline service can also be used to set up a data flow that reads from SQS and loads into Redshift. A pipeline would be created with an SQS source and a Redshift destination.
  • In all the above cases, the data would need to be first written to an S3 bucket from the Lambda/Glue job before using the COPYcommand to load it into Redshift. Redshift only supports loading data directly from S3 but not SQS. So an intermediate processing step is required to retrieve the messages from SQS and stage the data in S3.
profile picture
專家
已回答 2 個月前
AWS
支援工程師
已審閱 22 天前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南