Weird read performance

Hello,

I'm doing benchmark test of DocumentDB. Inserting documents with random payload and reading it back immediately by different consumer. Both writer and reader use multiple concurrent connection. DocumentDB is db4.r4.large in us-east-1; clients are nodejs using official mongodb npm client. Client run on EC2 in the same region and VPC.

The problem is that read timing has unexpected dependency on document size. up to about 4KB read takes < 4ms; Documents around 100KB are consumed within 7-8 ms; however for documents of around 8-16KB consumption takes 40ms on average (max is 70-80 ms, and I think this is the problem).

This is 100% reproducible.

Edit: to me it looks like not tcp_nodelay from server side?

Edit2: I don't have histogram, but max of read time seems to confirm nodelay theory. Max for 1-4KB is 20-30 ms; max for 8KB is 70-80 ms; max for 100KB is 6-8 ms!!! I.e. for 100 KB payload variance is very low, while for small payloads it is huge.

Edited by: Alexey on Jan 28, 2019 3:29 AM

Edited by: Alexey on Jan 28, 2019 3:33 AM