By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Can I use packetbeat or some other mechanism to monitor search queries hitting my AWS OpenSearch cluster?

0

Hi, I'm trying to find a good way to monitor search queries hitting our AWS OpenSearch cluster.

Use case: We have dozens of different applications that send queries to the cluster, some in response to user requests, some based on a cron schedule or other background activities. I need a way to analyze the volume of different types of queries being processed, average elapsed time for each type of query (e.g. by index and/or some normalized query pattern).

The most promising option I've found searching on the web for this problem is packetbeat. See https://www.elastic.co/blog/monitoring-the-search-queries for details on this setup. So, I'm wondering:

  • Is it possible to monitor AWS OpenSearch cluster using packetbeat?
  • Is there an alternate or better way to achieve what I described above?

I'm aware of slow query logging. This is somewhat helpful but not sufficient for my needs.

We are currently running elasticsearch version 6.8, so ideally looking for a solution that is compatible with that version, but we are also planning to upgrade next year to a newer version so I am interested in solutions that work with newer versions as well. Thanks!

Topics
ServerlessAnalytics
Tags
Amazon OpenSearch Service
Language
English
satwood
asked 7 months ago256 views
1 Answer
1

Hello,

On checking the link which you have shared as per my understanding the solution cannot be implemented in case of AWS OpenSearch service. This is because in start sniffing step I can see the instructions of installing Packetbeat to the nodes of production cluster which is unfortunately not possible due to managed nature of OpenSearch service.

Unfortunately, other than slow logs currently I do not see any other AWS recommended solution for monitoring slow queries. There are metrics like searchLatency in cloudwatch metrics which in combination of slow logs may help.

[+] Monitoring OpenSearch cluster metrics with Amazon CloudWatch - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html

AWS
SUPPORT ENGINEER
Aman_A
answered 7 months ago
  • satwood
    6 months ago

    Thank you. This was my suspicion as well. I might look into feasibility of adding some instrumentation on the client side and/or sending requests through a proxy server where I could install packetbeat.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content