Can I load data to redshift serverless directly fom SQS

0

I have some json files being loaded to s3 and the events are being queued to sqs, I need to copy the information to redshift, I want to know if there is a way to load it directly from the sqs or if I must have a lambda or glue service running in the middle to retrieve the sqs.

2 Risposte
0

You're correct - you need some "middle" service that does the data ingest for you. Glue and Lambda (in that order) are probably the right things to look at but there are other ways such as running the data import in a container or on an EC2 instance.

profile pictureAWS
ESPERTO
con risposta 2 mesi fa
0

You definitely have several optiions:

  • You can use AWS Lambda to retrieve messages from the SQS queue. The Lambda function would then load the data into Redshift using the COPY command.
  • You can use AWS Glue to orchestrate the data flow. A Glue trigger would be configured to watch the SQS queue. When a message is received, Glue would retrieve it and use a Glue job to load the data into Redshift.
  • The AWS Data Pipeline service can also be used to set up a data flow that reads from SQS and loads into Redshift. A pipeline would be created with an SQS source and a Redshift destination.
  • In all the above cases, the data would need to be first written to an S3 bucket from the Lambda/Glue job before using the COPYcommand to load it into Redshift. Redshift only supports loading data directly from S3 but not SQS. So an intermediate processing step is required to retrieve the messages from SQS and stage the data in S3.
profile picture
ESPERTO
con risposta 2 mesi fa
AWS
TECNICO DI SUPPORTO
verificato 22 giorni fa

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande