Throttling a Client/Index from OpenSearch (ElasticSearch)
I have the following setup for logs:
- Fargate Service -> Firelens/Fluent Bit -> OpenSearch (ElasticSearch) Cluster or
- Fargate Service -> Cloudwatch -> OpenSearch (ElasticSearch) Cluster
I know about Fluent Bit Throttle, but that is on client side. What would be good mechanism on OpenSearch (ElasticSearch) Cluster side to throttle that Client/Index if he pushes to much data?
OpenSearch throttles by default when the respective threadpool queues are full. High threadpool queues indicate the that the incoming request rate is greater than what the cluster is provisioned to handle.
If you're going beyond this, you would have to setup a feedback mechanism by considering metrics such as
JVMMemoryPressure, etc. You can set a CloudWatch alarm over the metrics to notify you via SNS as well as configure SNS to invoke a Lambda function to reduce the rate on the client side as required.
The client might be in another account. The Lambda should execute there and reduce the rate there or should is there a way to to set some rule in OpenSearch to limit and incoming DataStream?
Throttling a Client/Index from OpenSearch (ElasticSearch)Accepted Answer
Impact to AWS Elasticsearch on the licensing change of Elasticsearch(elastic.co)Accepted Answerasked a year ago
Fluent Bit Logs, Kinesis vs OpenSearch (ElasticSearch) Directly
OpenSearch [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Eventasked 2 months ago
AWS OpenSearch Service got error after enable auto-tuneasked 12 days ago
Log Subscription Filter To Opensearchasked 6 months ago
Why does OpenSearch restrict the /_security endpoint?asked 4 months ago
What features of Open Distro is supported by ElasticSearch?Accepted AnswerMODERATORasked 3 years ago
Which foreign tokenizers does OpenSearch support?Accepted Answerasked 13 days ago
OpenSearch SNS Destination Support for Alertingasked 4 months ago