How to invoke Lambda using S3 Report Config file

0

I'm looking to use S3 Batch Operations to load historical log data that is stored in a single S3 bucket into a Redshift table. I created a report config file for the S3 bucket and have questions about how to do the following:

  1. How to query the S3 inventory configuration file for specific S3 objects and only send those specific objects to a Lambda which then puts the objects into a Datastream. Can I send the results of Athena query to the Lambda?
  2. Can the S3 Batch Operations send objects into a Datastream which will then forward them to Redshift?
bdoucet
gefragt vor einem Jahr250 Aufrufe
1 Antwort
0

Q1. You can follow below steps to achieve your query of "query the S3 inventory configuration file for specific S3 objects and only send those specific objects to a Lambda which then puts the objects into a Datastream."

  1. To create an inventory for a S3 bucket and store that inventory in a destination S3 bucket, please follow the steps mentioned in the document below https://linuxhint.com/create-inventory-configuration-s3-bucket/

  2. In order to query the inventory file for specific S3 objects from destination S3 bucket using Athena, you may refer the following https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html

  3. Once you have queried the desired objects store the resulting output file from Athena in a S3 bucket as described in below reference document https://docs.aws.amazon.com/athena/latest/ug/querying.html

  4. After you have stored the results file of Athena to a S3 bucket, you can use that file to invoke your Lambda function, please refer to the below reference document https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

  5. To send the data to data stream from Lambda you can follow the documentation https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html

Q2. To send S3 Batch Operations objects into a Data stream which will then forward them to Redshift

You may use S3 batch operations to invoke a Lambda function which will send it to the Data stream or Redshift based on your requirement. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-invoke-lambda.html

AWS
beantwortet vor einem Jahr

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen