Upgrade Elasticsearch cluster size and use 2 subnet ids.

• If we upgraded to 2 instances with a larger instance type, such as m5.large.elasticsearch.

During the upgrade, (I believe it is blue/green upgrade), we can still call API to insert documents and the inserts during upgrade will still be available after upgrade. Am I correct?

Yes you can index documents and the data would be available but there could be some performance degradation so AWS advises to run it during lean traffic hours. There is a chance of minimal downtime also during blue green deployment

Within same subnet, upgrading to more nodes will balance part of indexes and documents to 2nd node which provides performance enhancement due to load balance. Am I correct? Or actually the 2nd node will contain only replica shards from 1st node - which means query will improve but insert will not?

With two nodes and setting of one replica, you'll have an overall mix of primary and replicas on each node but not the primary and replica of the same shard on same node

With two nodes and no replicas, you'll have both nodes storing some documents / primary shards

• If we used 2 subnets and 2 nodes

will each subnet get one node? If so, will one become the total replica (only containing replica shards to ensure fail safe) of the other so queries/inserts/updates performance will not be improved?

yes each subnet would have a node. About replica assignment, each node would have a mix of primary and replica shards

if upgrading to 2 nodes, will the EBS volumn size be shared by these 2 nodes (eahch get 256 or just a shared ESB between 2 nodes?), or each node get 512?

Each node gets 512GB

Does version 5.3 even support dedicated master nodes?

Yes you can edit the cluster config to enable master nodes and it would trigger a Blue/Green deployment.

Generally indexing speed improves when you have more nodes and search improves with higher replica. But then in practise there are number of other factors which might impact the performance

Useful document about config changes - https://docs.aws.amazon.com/opensearch-service/latest/developerguide/managedomains-configuration-changes.html

-Syd