Opensearch Serverless - Slow and fails to index

0

I created a simple index with the following schema (in a search collection):

PUT musers1
{
    "mappings": {
        "properties": {
            "uuid":            { "type" : "keyword" },
            "ext_id":          { "type" : "keyword" },
            "last_name":       { "type" : "text", "fields": { "raw": { "type":  "keyword" }} },
            "first_name":      { "type" : "text", "fields": { "raw": { "type":  "keyword" }} },
            "primary_location":{ "type" : "long" },
            "created_date":    { "type" : "long" },
            "permissions":     { "type" : "keyword" },
            "roles":           { "type" : "keyword" }
        }
    }
}

Then I attempted to load 73 records in the dashboard (subset of data):

POST _bulk
{"index":{"_index":"musers1","_id":"14399575-cb41-4d33-4ee2-9a69c90a18af"}}
{"created_date":1464369890,"ext_id":"1111111","first_name":"Demo","last_name":"User1","primary_location":481549029373652992,"roles":["MEMBER"],"uuid":"14399575-cb41-4d33-4ee2-9a69c90a18af"}
{"index":{"_index":"musers1","_id":"f9644fb8-e790-429a-4965-9e18be4b8d8f"}}
{"created_date":1464369890,"ext_id":"2222222","first_name":"Demo","last_name":"User2","primary_location":481549029373652992,"roles":["MEMBER"],"uuid":"f9644fb8-e790-429a-4965-9e18be4b8d8f"}
{"index":{"_index":"musers1","_id":"4a621fc9-dc93-4934-756d-450412ca2f7a"}}
{"created_date":1607376361,"ext_id":"63844","first_name":"Demo","last_name":"User3","primary_location":1183110257241493504,"roles":["MEMBER","FT-FLOOR"],"uuid":"4a621fc9-dc93-4934-756d-450412ca2f7a"}
...

And I got back the response 4 minutes later!

{
  "took": 251331,
  "errors": true,
  "items": [
...

In the end, there were 64 records indexed, and the other nine failed with the error Timeout while processing request.

Also, I loaded 2000 records (slowly and painfully) yesterday into a different index and ran a few searches against it. I ended up with 5 Search and 5 Index OCUs. These OCUs have remained since yesterday. Is it normal to have 10 OCUs active when there is no indexing or search activity?

Is it normal for 73 records to fail 12% of the time and take 4 minutes? That's 18 records per minute. If I had 20 million records to index, (I have more than that actually) it would take over 2 years to index!

I hope I am doing something wrong, because this seems too slow and too expensive...

Any ideas? Z

質問済み 2年前1130ビュー
1回答
0

When you create your first collection, OpenSearch Serverless instantiates two OCUs—one for indexing and one for search. These OCUs exist even when there's no activity on any collection endpoints. All subsequent collections share these OCUs. When you create additional collections in the same account, OpenSearch Serverless only adds additional OCUs for search and ingest as needed to support the collections, according to the capacity limits that you specify. Capacity does scale back down as your compute usage decreases. [1]

If you are facing issue with OCUs not scaling down and latency while ingesting records, recommend you to kindly open a case with AWS Premium Support team so that they can have a better look at the resources.

[1] https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-overview.html#serverless-process

AWS
サポートエンジニア
回答済み 2年前
  • I have opened a ticket and I was informed that 6 index and 6 search OCUs were not enough to index 73 records successfully.

    I was hoping for a different answer from the users of the forum as this seems excessive.

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ