Can I load data to redshift serverless directly fom SQS

0

I have some json files being loaded to s3 and the events are being queued to sqs, I need to copy the information to redshift, I want to know if there is a way to load it directly from the sqs or if I must have a lambda or glue service running in the middle to retrieve the sqs.

2回答
0

You're correct - you need some "middle" service that does the data ingest for you. Glue and Lambda (in that order) are probably the right things to look at but there are other ways such as running the data import in a container or on an EC2 instance.

profile pictureAWS
エキスパート
回答済み 2ヶ月前
0

You definitely have several optiions:

  • You can use AWS Lambda to retrieve messages from the SQS queue. The Lambda function would then load the data into Redshift using the COPY command.
  • You can use AWS Glue to orchestrate the data flow. A Glue trigger would be configured to watch the SQS queue. When a message is received, Glue would retrieve it and use a Glue job to load the data into Redshift.
  • The AWS Data Pipeline service can also be used to set up a data flow that reads from SQS and loads into Redshift. A pipeline would be created with an SQS source and a Redshift destination.
  • In all the above cases, the data would need to be first written to an S3 bucket from the Lambda/Glue job before using the COPYcommand to load it into Redshift. Redshift only supports loading data directly from S3 but not SQS. So an intermediate processing step is required to retrieve the messages from SQS and stage the data in S3.
profile picture
エキスパート
回答済み 2ヶ月前
AWS
サポートエンジニア
レビュー済み 22日前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ