Pushing CloudTrail logs to ElasticSearch via SQS and Lambda

When invoking a Lambda function from a CloudWatch Logs subscription you cannot directly control messaging batching but there are a few options.

Option 1: You could configure reserved concurrency to limit the number of concurrent instances of the function that can be invoked. Whether or not this achieves your goal depends on the average request duration and requests per second, read more about Lambda concurrency. If requests from CloudWatch Logs are throttled, the events are returned to a queue and retried for up to 6 hours (by default); you can configure a SQS queue as a dead-letter queue to hold failed events. The Lambda function could then be invoked by SQS. This could be a bit tricky to configure properly without A) creating a large backlog of unprocessed events in SQS if concurrency is always used by the CloudWatch logs invocation, or B) creating a separate function to process the SQS queue

Option 2: You could create a dummy Lambda function with reserved concurrency set to 0 and the failure destination/dead-letter queue set to SQS. CloudWatch logs would fail to invoke this dummy function (no concurrency available) so the event data would be sent to SQS without retries. The "real" Lambda function could then be invoked by SQS. In this scenario there are several options to control batching and concurrency including batch size (number of records to send to the function from SQS), batch window (amount of time to gather records before invoking the function), and maximum concurrency (maximum number of concurrent functions that SQS can invoke), see Using Lambda with Amazon SQS for more details.

Option 3: If you're only sending CloudTrail logs to CloudWatch for the Lambda subscription filter you could eliminate CloudWatch and use SNS notifications for CloudTrail with a SQS queue subscribed to the SNS topic. As CloudTrail writes logs files to the S3 bucket, a notification is sent to SNS — the notification includes the S3 bucket name and object key(s) of the log file(s). SNS then sends the message to the subscribed SQS queue; SQS would then trigger the Lambda function. In this scenario there are several options to control batching and concurrency including batch size (number of records to send to the function from SQS), batch window (amount of time to gather records before invoking the function), and maximum concurrency (maximum number of concurrent functions that SQS can invoke), see Using Lambda with Amazon SQS for more details. It's important to note that in this scenario, you'd have to retrieve the files from S3 then decompress and process them. Also, each SQS message could contain multiple files and each file could multiple CloudTrail log records.