Lambda log to Elastic Search. Not using CloudWatch!

Yes, it is possible to send Lambda logs directly to Elasticsearch without using CloudWatch. This can be achieved using the Lambda Telemetry API and a custom Lambda extension.

The Lambda Telemetry API allows you to send logs, metrics, and traces directly from your Lambda functions to third-party monitoring and observability tools, including Elasticsearch. By using this API in combination with a custom Lambda extension, you can bypass CloudWatch entirely and stream your logs directly to your Elasticsearch cluster.

Here's how you can implement this:

1. Create a custom Lambda extension that uses the Lambda Telemetry API to capture logs and telemetry data from your Lambda function.
2. Configure your extension to send this data directly to your personal Elasticsearch servers.
3. Ensure your Lambda function has the necessary permissions to use the Telemetry API and send data to your Elasticsearch servers.

This approach offers several benefits:

• Real-time log streaming and analysis
• Reduced latency compared to using CloudWatch as an intermediary
• Flexibility to customize the data sent to Elasticsearch
• Integration with Elasticsearch's powerful search, analytics, and visualization capabilities

Regarding costs, by bypassing CloudWatch, you'll avoid CloudWatch Logs charges. The main AWS cost to consider will be data transfer out of AWS to your personal Elasticsearch servers. AWS charges for data transfer from Lambda to the internet, which would apply when sending logs to your external servers. The exact cost depends on your region and the amount of data transferred.

To estimate costs, you'll need to:

1. Calculate the volume of log data your Lambda functions generate.
2. Check the AWS pricing for data transfer out in your region.

Keep in mind that while this approach can save on CloudWatch costs, it requires more setup and maintenance on your part. You'll need to manage your own Elasticsearch cluster, ensure it can handle the log volume, and implement appropriate security measures for data transfer and storage.
Sources
Send telemetry data from AWS Lambda to OpenSearch for real-time analytics and visualization - AWS Prescriptive Guidance
Working with Lambda function logs - AWS Lambda