How to invoke Lambda using S3 Report Config file

0

I'm looking to use S3 Batch Operations to load historical log data that is stored in a single S3 bucket into a Redshift table. I created a report config file for the S3 bucket and have questions about how to do the following:

  1. How to query the S3 inventory configuration file for specific S3 objects and only send those specific objects to a Lambda which then puts the objects into a Datastream. Can I send the results of Athena query to the Lambda?
  2. Can the S3 Batch Operations send objects into a Datastream which will then forward them to Redshift?
bdoucet
asked a year ago243 views
1 Answer
0

Q1. You can follow below steps to achieve your query of "query the S3 inventory configuration file for specific S3 objects and only send those specific objects to a Lambda which then puts the objects into a Datastream."

  1. To create an inventory for a S3 bucket and store that inventory in a destination S3 bucket, please follow the steps mentioned in the document below https://linuxhint.com/create-inventory-configuration-s3-bucket/

  2. In order to query the inventory file for specific S3 objects from destination S3 bucket using Athena, you may refer the following https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-inventory-athena-query.html

  3. Once you have queried the desired objects store the resulting output file from Athena in a S3 bucket as described in below reference document https://docs.aws.amazon.com/athena/latest/ug/querying.html

  4. After you have stored the results file of Athena to a S3 bucket, you can use that file to invoke your Lambda function, please refer to the below reference document https://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html

  5. To send the data to data stream from Lambda you can follow the documentation https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html

Q2. To send S3 Batch Operations objects into a Data stream which will then forward them to Redshift

You may use S3 batch operations to invoke a Lambda function which will send it to the Data stream or Redshift based on your requirement. https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-invoke-lambda.html

AWS
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions