Can I use packetbeat or some other mechanism to monitor search queries hitting my AWS OpenSearch cluster?

0

Hi, I'm trying to find a good way to monitor search queries hitting our AWS OpenSearch cluster.

Use case: We have dozens of different applications that send queries to the cluster, some in response to user requests, some based on a cron schedule or other background activities. I need a way to analyze the volume of different types of queries being processed, average elapsed time for each type of query (e.g. by index and/or some normalized query pattern).

The most promising option I've found searching on the web for this problem is packetbeat. See https://www.elastic.co/blog/monitoring-the-search-queries for details on this setup. So, I'm wondering:

  • Is it possible to monitor AWS OpenSearch cluster using packetbeat?
  • Is there an alternate or better way to achieve what I described above?

I'm aware of slow query logging. This is somewhat helpful but not sufficient for my needs.

We are currently running elasticsearch version 6.8, so ideally looking for a solution that is compatible with that version, but we are also planning to upgrade next year to a newer version so I am interested in solutions that work with newer versions as well. Thanks!

satwood
gefragt vor 7 Monaten265 Aufrufe
1 Antwort
1

Hello,

On checking the link which you have shared as per my understanding the solution cannot be implemented in case of AWS OpenSearch service. This is because in start sniffing step I can see the instructions of installing Packetbeat to the nodes of production cluster which is unfortunately not possible due to managed nature of OpenSearch service.

Unfortunately, other than slow logs currently I do not see any other AWS recommended solution for monitoring slow queries. There are metrics like searchLatency in cloudwatch metrics which in combination of slow logs may help.

[+] Monitoring OpenSearch cluster metrics with Amazon CloudWatch - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-cloudwatchmetrics.html

AWS
SUPPORT-TECHNIKER
Aman_A
beantwortet vor 7 Monaten
  • Thank you. This was my suspicion as well. I might look into feasibility of adding some instrumentation on the client side and/or sending requests through a proxy server where I could install packetbeat.

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen