Neptune full text search with OpenSearch Serverless issues

Question

We recently connected our Neptune cluster to OpenSearch Service and eventually, the data transfer finished successfully and we can perform queries with OpenSearch Service. 
Now we are trying to do the same via OpenSearch Serverless so that we can benefit from the autoscallability part. But, we ran into some problems when performing the data transfer from Neptune to OS Serverless. More specifically, 429 error
BulkIndexError: ('234 document(s) failed to index.', [{'update': {'_index': 'amazon_neptune', 'status': 429, 'error': {'type': 'circuit_breaking_exception', 'reason': 'rejected execution of primary operation [throttled]', 'bytes_wanted': 0, 'bytes_limit': 0, 'durability': 'TRANSIENT'}

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/serverless-scaling.html

Looking at the index settings, I saw that the number of shards is just 2, while in the poller function created by the CF stack (https://docs.aws.amazon.com/neptune/latest/userguide/full-text-search.html) I clearly specified a shard count of 5. I saw that the OCUs on the indexing part did get up to 5 and the limit was set to 10 by default, but it still failed.

Any idea how to solve this? I tried modifying the settings of the index via the OS API or directly creating it via that API, but it still doesn't allow a higher count of more than 2 shards.

One last question: is there any way of increasing the speed of ingestion? In the Lambda function logs, I see this "Executing 1 batch queries with batch size 2500 in parallel using 1 workers" Any way of increasing the number of workers? It took quite a long time to ingest data via OpeSearch Service as well.

Thank you.

Param Jani · Answer

Hello,

To answer your question, we would require details that are non-public information. Please open a support case with AWS using the following link:

https://console.aws.amazon.com/support/home#/case/create

If a support case has already been created please be assured that we will get back to you and assist you in the best way possible.

Thank you!

re:Post Agent · Answer

Thank you for providing details about your Neptune full-text search integration with OpenSearch Serverless. Let me address your concerns and provide some insights.

Regarding the 429 error you're encountering, this is typically related to throttling issues. The circuit breaking exception suggests that the OpenSearch Serverless cluster is reaching its capacity limits during the data transfer process.

The shard count discrepancy you've noticed (2 instead of 5) is likely due to OpenSearch Serverless managing sharding automatically. Unlike with managed OpenSearch Service, Serverless doesn't allow manual configuration of shard counts. The system determines the optimal number of shards based on the data volume and workload.

To address the ingestion issues:

1. Increase the OCU (OpenSearch Compute Unit) limit: While you mentioned the limit was set to 10 by default, you might want to consider increasing this further if your data volume is substantial.

2. Adjust batch size: The Lambda function is using a batch size of 2500. You could try reducing this to see if it helps with the throttling issues.

3. Gradual data transfer: Instead of transferring all data at once, consider breaking it into smaller chunks and transferring them incrementally.

4. Retry mechanism: Implement a retry mechanism with exponential backoff in your Lambda function to handle temporary throttling issues.

Regarding increasing the speed of ingestion, unfortunately, the number of workers used by the Lambda function is not directly configurable. However, you can try the following:

1. Increase Lambda function memory: This can indirectly increase CPU allocation and potentially improve processing speed.

2. Use multiple Lambda functions: Instead of relying on a single function, you could split the workload across multiple Lambda functions, each responsible for a subset of your data.

3. Optimize your graph data: Ensure your Neptune database is optimized, as this can impact the speed of data extraction.

4. Monitor and adjust: Keep a close eye on CloudWatch metrics for both Neptune and OpenSearch Serverless to identify bottlenecks and adjust your approach accordingly.

Remember that OpenSearch Serverless is designed to handle scaling automatically, so some of the manual optimizations available in the managed service may not be applicable. Focus on working within the Serverless model's constraints and leveraging its auto-scaling capabilities.

Lastly, if these suggestions don't resolve your issues, consider reaching out to AWS support for more personalized assistance, as they can provide insights based on your specific account and usage patterns.  
**Sources**  
[Amazon Neptune now supports for Amazon OpenSearch 2.3, 2.5, and Serverless](https://aws.amazon.com/about-aws/whats-new/2024/02/amazon-neptune-supports-opensearch-serverless/)  
[Full text search in Amazon Neptune using Amazon OpenSearch Service - Amazon Neptune](https://docs.aws.amazon.com/neptune/latest/userguide/full-text-search.html)

Neptune full text search with OpenSearch Serverless issues

Relevant content