OpenSearch - Incorrect Total Hits

Hi all,

in advance, sorry for the longer question. 🙃

We're dealing with a weird error where, in a response to a simple search operation, I'm getting an incorrect total hits value. A bit more detail in the following text.

We have three records and then we execute two queries.

So, this first query is basically "give me first 2 items in a list of all records".

GET local2_api-headless-cms-env-root-headless-cms-en-us-acosearchrecord-webiny/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "__type.keyword": "cms.entry.l"
          }
        },
        {
          "term": {
            "values.text@type.keyword": "page"
          }
        }
      ]
    }
  },
  "sort": {
    "createdOn": {
      "order": "asc",
      "unmapped_type": "date"
    }
  },
  "size": 3,
  "track_total_hits": true
}

Note that size is set to 3, simply because in our app code we check if there's a +1 item in the result. If so, that means the next page actually exists. Other than that, we don't do anything else with the third returned item in the result.

With this first query run against OS, we receive the correct response, where the 2nd item (out of three items in total) has its sort field set to 1699010027184. We then use that value in the following query to retrieve the second page:

GET local2_api-headless-cms-env-root-headless-cms-en-us-acosearchrecord-webiny/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "__type.keyword": "cms.entry.l"
          }
        },
        {
          "term": {
            "values.text@type.keyword": "page"
          }
        }
      ]
    }
  },
  "sort": {
    "createdOn": {
      "order": "asc",
      "unmapped_type": "date"
    }
  },
  "size": 3,
  "search_after": [
    "1699010027184"
  ],
  "track_total_hits": true
}

The result of the above query is the following:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 1,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "local2_api-headless-cms-env-root-headless-cms-en-us-acosearchrecord-webiny",
        "_id": "T#root#L#en-US#CMS#CME#wby-aco-page-c:L",
        "_score": null,
        "_source": {
          "webinyVersion": "0.0.0",
          "tenant": "root",
          "entryId": "wby-aco-page-c",
          "id": "wby-aco-page-c#0001",
          "modelId": "acoSearchRecord-webiny",
          "locale": "en-US",
          "createdOn": "2023-11-03T11:13:47.355Z",
          "savedOn": "2023-11-03T11:13:47.355Z",
          "createdBy": {
            "id": "12345678",
            "displayName": "John Doe",
            "type": "admin"
          },
          "ownedBy": {
            "id": "12345678",
            "displayName": "John Doe",
            "type": "admin"
          },
          "modifiedBy": null,
          "version": 1,
          "locked": false,
          "status": "draft",
          "values": {
            "text@type": "page",
            "text@title": "Page c",
            "text@content": "Lorem ipsum docet",
            "object@location": {
              "text@folderId": "6544d5e928eb040c1fee9e81#0001"
            },
            "object@data": {
              "text@someText": "A text which is searchable as well",
              "object@identity": {
                "text@id": "user-3",
                "text@displayName": "Janine Doe",
                "text@type": "admin"
              },
              "datetime@customCreatedOn": "2023-03-15T00:00:00.000Z",
              "number@customVersion": 2,
              "boolean@customLocked": false
            },
            "text@tags": [
              "page-tag3"
            ]
          },
          "location": {
            "folderId": "root"
          },
          "rawValues": {
            "object@location": {},
            "object@data": {
              "object@identity": {}
            }
          },
          "latest": true,
          "TYPE": "cms.entry.l",
          "__type": "cms.entry.l"
        },
        "sort": [
          1699010027355
        ]
      }
    ]
  }
}

So, what's the problem?

Well, despite the fact that OS did correctly return the 3rd item in the list, if you check the hits section, you can see it's incorrectly reporting 2 (instead of 3) as the total number of records:

   "total": {
      "value": 2,
      "relation": "eq"
    },

And then the question is: does anybody have a clue why this could be happening?

The thing is, we've had the same exact query running against the "old" ElasticSearch, and there all works as expected. The total hits number correctly says 3.

The only difference I noticed upon comparing the OS vs. ES results is that, with OS, in the _shards section, we have 1 shard marked as skipped:

  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 1,
    "failed": 0
  },

Could that maybe be the reason for the incorrect total hits number?

If so, anybody have an idea on why a shard would be skipped? After a bit of reading, I gathered that OS will skip a shard if it "knows" it doesn't contain the related data.

Ultimately, other than the skipped: 1 detail, I could not discover anything else that might potentially help me in resolving the issue. So, the next step was to ask here. ;)

Thank you!

UPDATE: One other piece of information I should share is that when running the same thing against a locally set up OS, all works. So it's just the AWS's OS that's causing the issue.