Usando AWS re:Post, accetti AWS re:Post Termini di utilizzo
AWS re:Postre:Post

Throttling a Client/Index from OpenSearch (ElasticSearch)

0

I have the following setup for logs:

  • Fargate Service -> Firelens/Fluent Bit -> OpenSearch (ElasticSearch) Cluster or
  • Fargate Service -> Cloudwatch -> OpenSearch (ElasticSearch) Cluster

I know about Fluent Bit Throttle, but that is on client side. What would be good mechanism on OpenSearch (ElasticSearch) Cluster side to throttle that Client/Index if he pushes to much data?

Argomenti
ServerlessAnalisi
Tag
Servizio Amazon OpenSearch
Lingua
English
Laszlo
posta 2 anni fa2399 visualizzazioni
1 Risposta
1
Risposta accettata

OpenSearch throttles by default when the respective threadpool queues are full. High threadpool queues indicate the that the incoming request rate is greater than what the cluster is provisioned to handle.

If you're going beyond this, you would have to setup a feedback mechanism by considering metrics such as CPUUtilization, JVMMemoryPressure, etc. You can set a CloudWatch alarm over the metrics to notify you via SNS as well as configure SNS to invoke a Lambda function to reduce the rate on the client side as required.

AWS
Ajay N
con risposta 2 anni fa
  • Laszlo
    2 anni fa

    The client might be in another account. The Lambda should execute there and reduce the rate there or should is there a way to to set some rule in OpenSearch to limit and incoming DataStream?

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande

Contenuto pertinente