Latency increase as number of clients increase.

Hi Team,

I am using Openserach-benchmark tool to benchmark my Opensearch Service cluster, Region: us-east-1.

Nodes: 6 (r6g.12xlarge instance 48vCPUs and 384GB memory)

Master nodes: 3

Shards: 158

EBS volume: 24TB GP3 type

0 replica.

~11B documents.

I observe that the service time metric is almost double if I double the number of clients keeping the target throughput constant. For ex.

num_clients, target_throughput, service time p99(p99.9)
35, 100, 121.986(140.48)
75, 100, 236.562(268.817)

The CPU, disk utilization of the cluster is well under the limit. Also, I tried benchmarking with a machine 96vCPUs. I still see the same pattern. I’m not able to reason this behavior. From Opensearch service point of view, irrespective of the number of clients generating the same load, the service should handle the same right? Am I missing something? Please let me know if you need more info. from my side. Thanks