Opensearch memory utilization is consistently high at 99%, and CPU utilization occasionally reaches 100% (about 4 times per day).

Based on the information provided, there are a few factors contributing to your high memory and CPU utilization:

1. Memory Utilization: The consistently high memory utilization (99%) suggests that your current instance type (r7g.4xlarge.search) may not have enough memory to handle your workload efficiently. This could be due to the size of your indices, especially indexB which is quite large at 343.2 GiB. Elasticsearch uses a significant amount of memory for caching and query processing, so it's not uncommon to see high memory usage.

2. CPU Utilization: The occasional spikes to 100% CPU utilization (about 4 times per day) could be related to various factors:

• The hourly cron job to save new data might be causing these spikes.
• Complex queries or aggregations running on your large indices.
• Indexing operations, especially if they coincide with high query loads.

3. Instance Type and Node Count: While the r7g.4xlarge.search instance type is memory-optimized, which is good for Elasticsearch workloads, you may need to consider either increasing the instance size or adding more data nodes to your cluster.

Recommendations:

1. Increase Memory: Consider upgrading to a larger instance type with more memory, such as r7g.8xlarge.search, which would double your memory capacity.

2. Add More Nodes: Instead of upgrading the instance type, you could add one or two more data nodes to distribute the load and provide more overall memory and CPU capacity to your cluster.

3. Optimize Indices: Review your index settings and shard allocation. With only two data nodes and 5 shards per index, you might benefit from adjusting the number of shards or their allocation across nodes.

4. Monitor and Analyze: Use Elasticsearch's monitoring tools to identify specific queries or operations causing CPU spikes. This can help in optimizing those particular processes.

5. Caching Strategy: Evaluate and possibly adjust your caching strategy to make better use of available memory.

6. Schedule Maintenance: Consider scheduling resource-intensive operations (like the hourly cron job) during off-peak hours to minimize impact on performance.

Regarding cost management, while you should aim to reduce costs, running Elasticsearch nodes with very high memory utilization can impact performance and stability. The right balance would be to have enough headroom (usually 10-20% free memory) for optimal performance while not over-provisioning resources.

Consider starting with either adding one more node of the same type or upgrading to the next larger instance size. Monitor the performance and utilization after making changes to find the optimal configuration for your workload and budget.

Sources
Amazon EC2 High Memory Instances – Amazon Web Services (AWS)
would like to remove or reduce 100% cpu usage from VPC or EC2, pls provide solution towards | AWS re:Post
Amazon ECS service utilization metrics - Amazon Elastic Container Service