By using AWS re:Post, you agree to the Terms of Use

Log Subscription Filter To Opensearch

0

I have enabled a CloudWatch log subscription filter which auto generates Lambda function. The Lambda function is supposed to stream log data to Opensearch domain. Does the Lambda function create a new index and indices in Opensearch? Currently, I only see the system indexes present in Opensearch domain.

1 Answer
0

You're right. The lambda function will create indices with name like cwl-YYYY.MM.DD. You can change the naming pattern by modifying the code in the lambda function where the variable indexName is being declared.

Since you are using OpenSearch for indexing log data, consider UltraWarm which provides a low-cost storage tier for older read-only data like logs which you might query less frequently.

Related blog post - https://aws.amazon.com/blogs/aws/general-availability-of-ultrawarm-for-amazon-elasticsearch-service/

answered 10 months ago
  • I thank you for the great feedback. Somewhere there's a disconnect between the auto Lambda function and Opensearch. From CloudWatch metrics, I can see the Lambda function is being called and new logs are being processed. In Opensearch, document metrics have increased, but I don't see any additional indexes or indices when I access the Opensearch dashboard from the browser. Documentation is very limited on this functionality. Would you happen to have detailed documentation for a complete end to end log subscription filter stream to Opensearch.

  • To see a list of indices currently in your OpenSearch domain, you can try the following GET request from browser or postman that can access your OpenSearch domain : https://<your_os_domain_endpoint_url>/*?expand_wildcards=all . Replace <your_os_domain_endpoint_url> with your domain endpoint found in AWS Console.

    Instead of *, you can use cwl* in the request which will list all indices starting with cwl i.e https://<your_os_domain_endpoint_url>/cwl*?expand_wildcards=all .

    If the new index is not listed, it could be the case that there is some error indexing the documents at lambda whether due to the access policy defined for your OpenSearch domain or other reasons. You could check the lambda logs in Cloudwatch for any errors. Some common issues are document here - https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-troubleshoot-cloudwatch-logs/

    I couldn't find an end-to-end guide other than the one in this documentation - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html. However I did find this post about setting up publishing RDS logs to CloudWatch and streaming that to OpenSearch which documents most of the steps to follow - https://aws.amazon.com/blogs/database/analyze-postgresql-logs-with-amazon-elasticsearch-service/ .

    Hope that's useful with the issue you are facing.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions