1 Answer
- Newest
- Most votes
- Most comments
0
You're right. The lambda function will create indices with name like cwl-YYYY.MM.DD. You can change the naming pattern by modifying the code in the lambda function where the variable indexName is being declared.
Since you are using OpenSearch for indexing log data, consider UltraWarm which provides a low-cost storage tier for older read-only data like logs which you might query less frequently.
Related blog post - https://aws.amazon.com/blogs/aws/general-availability-of-ultrawarm-for-amazon-elasticsearch-service/
answered 2 years ago
Relevant content
- Accepted Answerasked 10 months ago
AWS OFFICIALUpdated a year ago- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 months ago
I thank you for the great feedback. Somewhere there's a disconnect between the auto Lambda function and Opensearch. From CloudWatch metrics, I can see the Lambda function is being called and new logs are being processed. In Opensearch, document metrics have increased, but I don't see any additional indexes or indices when I access the Opensearch dashboard from the browser. Documentation is very limited on this functionality. Would you happen to have detailed documentation for a complete end to end log subscription filter stream to Opensearch.
To see a list of indices currently in your OpenSearch domain, you can try the following GET request from browser or postman that can access your OpenSearch domain : https://<your_os_domain_endpoint_url>/*?expand_wildcards=all . Replace <your_os_domain_endpoint_url> with your domain endpoint found in AWS Console.
Instead of *, you can use cwl* in the request which will list all indices starting with cwl i.e https://<your_os_domain_endpoint_url>/cwl*?expand_wildcards=all .
If the new index is not listed, it could be the case that there is some error indexing the documents at lambda whether due to the access policy defined for your OpenSearch domain or other reasons. You could check the lambda logs in Cloudwatch for any errors. Some common issues are document here - https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-troubleshoot-cloudwatch-logs/
I couldn't find an end-to-end guide other than the one in this documentation - https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_OpenSearch_Stream.html. However I did find this post about setting up publishing RDS logs to CloudWatch and streaming that to OpenSearch which documents most of the steps to follow - https://aws.amazon.com/blogs/database/analyze-postgresql-logs-with-amazon-elasticsearch-service/ .
Hope that's useful with the issue you are facing.