Can I load data to redshift serverless directly fom SQS

0

I have some json files being loaded to s3 and the events are being queued to sqs, I need to copy the information to redshift, I want to know if there is a way to load it directly from the sqs or if I must have a lambda or glue service running in the middle to retrieve the sqs.

2 回答
0

You're correct - you need some "middle" service that does the data ingest for you. Glue and Lambda (in that order) are probably the right things to look at but there are other ways such as running the data import in a container or on an EC2 instance.

profile pictureAWS
专家
已回答 2 个月前
0

You definitely have several optiions:

  • You can use AWS Lambda to retrieve messages from the SQS queue. The Lambda function would then load the data into Redshift using the COPY command.
  • You can use AWS Glue to orchestrate the data flow. A Glue trigger would be configured to watch the SQS queue. When a message is received, Glue would retrieve it and use a Glue job to load the data into Redshift.
  • The AWS Data Pipeline service can also be used to set up a data flow that reads from SQS and loads into Redshift. A pipeline would be created with an SQS source and a Redshift destination.
  • In all the above cases, the data would need to be first written to an S3 bucket from the Lambda/Glue job before using the COPYcommand to load it into Redshift. Redshift only supports loading data directly from S3 but not SQS. So an intermediate processing step is required to retrieve the messages from SQS and stage the data in S3.
profile picture
专家
已回答 2 个月前
AWS
支持工程师
已审核 22 天前

您未登录。 登录 发布回答。

一个好的回答可以清楚地解答问题和提供建设性反馈,并能促进提问者的职业发展。

回答问题的准则