AWS OpenSearch Heap Size increase

0

I am working with an OpenSearch service cluster that during reindexing is running out of memory and the writes get blocked.

I see that the default limit for the heap is 32GB but my nodes have 10 times that. Is there a way to increase the heap size? Should this be done through a service quote increase? I couldn't find the right quota to select.

preguntada hace un mes76 visualizaciones
1 Respuesta
0

The heap size limit for Amazon OpenSearch Service is indeed 32 GiB, and this is a hard limit that cannot be increased through a service quota request. This limit is in place regardless of the total memory available on your nodes.

However, there are a few strategies you can consider to address your out-of-memory issues during reindexing:

  1. Auto-Tune: Amazon OpenSearch Service offers an Auto-Tune feature that can help optimize your cluster's performance. It can suggest and implement changes to various settings, including JVM heap size (up to the 32 GiB limit) and JVM young generation settings. These optimizations can potentially help with memory management during intensive operations like reindexing.

  2. Queue Size Adjustment: Auto-Tune can also adjust queue sizes. By default, the search queue size is 1000 and the write queue size is 10000. Auto-Tune can scale these queues if additional heap is available, which might help manage the load during reindexing.

  3. Scaling: If you're consistently running into memory issues, you might need to consider scaling your cluster. This could involve adding more nodes or using larger instance types to distribute the workload.

  4. Optimize Reindexing Process: Review your reindexing process to see if it can be optimized. This might involve breaking the reindexing into smaller batches or adjusting your indexing settings.

  5. Monitor and Adjust JVM Settings: While you can't exceed the 32 GiB heap limit, you can monitor your cluster's performance and adjust the percentage of RAM allocated to the JVM heap (up to the 32 GiB limit). By default, OpenSearch Service uses 50% of an instance's RAM for the JVM heap.

Remember, increasing the heap size isn't always the best solution for performance issues. Sometimes, other optimizations or architectural changes can be more effective in addressing memory-related problems.
Sources
Amazon OpenSearch Service quotas - Amazon OpenSearch Service
Auto-Tune for Amazon OpenSearch Service - Amazon OpenSearch Service

profile picture
respondido hace un mes

No has iniciado sesión. Iniciar sesión para publicar una respuesta.

Una buena respuesta responde claramente a la pregunta, proporciona comentarios constructivos y fomenta el crecimiento profesional en la persona que hace la pregunta.

Pautas para responder preguntas