By using AWS re:Post, you agree to the Terms of Use
/How to perform scroll / scroll not working on Amazon Elasticsearch/

How to perform scroll / scroll not working on Amazon Elasticsearch

0

We used this guide: https://opensearch.org/docs/latest/opensearch/rest-api/scroll/, and the amazon elasticsearch docs: https://docs.aws.amazon.com/opensearch-service/latest/developerguide/supported-operations.html to try and find out how to perform scroll search on amazon elasticsearch. We recently switched from elasticsearch on one of our local servers to amazon elasticsearch. We came to realize that scroll performs differently. See below a snippet on how we now try to perform scroll (in python3): params = {"size": 5000, "q": query, "scroll": "5m"} resp = requests.get(ELASTICSEARCH_URL + "/_search", params=params, auth=auth) resp.status code: 200 scroll_id = resp.json().get("_scroll_id") In 23: resp = requests.get(ELASTICSEARCH_URL + "_search/scroll", params={"scro ...: ll": "5m", "scroll_id": scroll_id}, auth=auth) In 24: resp Out24: <Response 400> In 25: resp._content Out25: b'{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request /users/_search/scroll contains unrecognized parameters: scroll, scroll_id"}],"type":"illegal_argument_exception","reason":"request /users/_search/scroll contains unrecognized parameters: scroll, scroll_id"},"status":400}'

We've tried about all the variations, remove "/"'s or tweaking params, but getting this same error.

Thank you in advance.

asked 5 months ago37 views
1 Answers
0

Syntax for scroll request: GET _search/scroll/<scroll-id>

https://opensearch.org/docs/latest/opensearch/rest-api/scroll/

When running the scroll request for the first time, we use the index name. Below is an example:

GET shakespeare/_search?scroll=10m { "size": 10000 }

This will return scroll id:

"_scroll_id" : "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAUWdmpUZDhnRFBUcWFtV21nMmFwUGJEQQ=="

From the second time use only _search/scroll/<scrollid>. we don't pass index name the second time

sample: GET _search/scroll { "scroll": "10m", "scroll_id": "DXF1ZXJ5QW5kRmV0Y2gBAAAAAAAAAAUWdmpUZDhnRFBUcWFtV21nMmFwUGJEQQ==" }

Note: Once the scroll id expires, scroll request will start failing. Please consider starting a new scroll.

params = {"size": 5000, "q": query, "scroll": "5m"} resp = requests.get(ELASTICSEARCH_URL + "/_search", params=params, auth=auth) resp.status code: 200 scroll_id = resp.json().get("_scroll_id")

Above Query works fine on versions 7.10, 7.9, 7.7 and 7.4. Since python says illegal argument exception, version of ES/OS is required to replicate the issue using the same python client.

SUPPORT ENGINEER
answered 5 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions