How do I resolve the error "Cache of region boundaries are out of date" when running an HBase read-replica Amazon EMR cluster with Phoenix?

Short description

Phoenix tries to connect to the hbase:meta table by default. However, because the hbase:meta table belongs to the primary cluster, Phoenix can't to connect the read-replica cluster. To resolve this problem, modify hbase-site.xml to point to the hbase:meta_cluster-id table that belongs to the HBase read-replica cluster.

Resolution

Before you begin, confirm that Phoenix is installed on the primary cluster as well as the read-replica cluster. Phoenix can't connect to HBase from a read-replica cluster if Phoenix isn't installed on the primary cluster.

On a running cluster

1.    Add the following configurations to the HBase configuration file ( /etc/phoenix/conf/hbase-site.xml) on the master node. Replace cluster-id with the ID of your read-replica cluster.

<property>
   <name>hbase.balancer.tablesOnMaster</name>
   <value>hbase:meta</value>
</property>
<property>
   <name>hbase.meta.table.suffix</name>
   <value>cluster-id</value>
</property>

2.    Restart the Phoenix service:

For Amazon EMR release versions 5.29 and earlier:

sudo stop phoenix-queryserver
sudo start phoenix-queryserver

For Amazon EMR release versions 5.30 and later:

sudo systemctl stop phoenix-queryserver.service
sudo systemctl start phoenix-queryserver.service

In Amazon EMR release versions 5.21.0 and later, you can also make these configuration changes by overriding the cluster configuration for the master instance group:

1.    Open the Amazon EMR console.

2.    In the cluster list, choose the active read-replica cluster that you want to reconfigure.

3.    Open the cluster details page for the cluster and go to Configurations tab.

4.    In the Filter dropdown list, choose the master instance group.

5.    In Reconfigure dropdown list, choose either Edit in table.

6.    Choose Add configuration, and then add the following two configurations:

Classification: phoenix-hbase-site
Property: hbase.balancer.tablesOnMaster
Value: hbase:meta

Classification: phoenix-hbase-site
Property: hbase.meta.table.suffix
Value: ${emr.clusterId}

7.    Choose Save changes.

For more information about this process, see Reconfigure an instance group in the console.

On a new cluster

Add a configuration object similar to the following when you launch a cluster using Amazon EMR release version 4.6.0 or later:

[
  {
    "Classification": "phoenix-hbase-site", 
    "Configurations": [
      ], 
    "Properties": {
      "hbase.balancer.tablesOnMaster" : "hbase:meta",
      "hbase.meta.table.suffix" : "${emr.clusterId}"
      }
  },
  {
    "Classification": "hbase-site", 
    "Configurations": [
      ], 
    "Properties": {
      "hbase.meta.table.suffix" : "${emr.clusterId}"
      }
  }
]

Related information

HBase on Amazon S3 (Amazon S3 storage mode)

Apache Phoenix