EMR Hive Meta Database on RDS Upgrade Issue


My customer is using AWS EMR and is storing all the Hive meta data on an external RDS instance, using MySQL 5.7.* And since MySQL 5.7 is running out of its lifecycle, we are pushing them to upgrade the RDS instance to MySQL 8.0. However, my customer is very concerned in regarding the RDS upgrade. So the basic question is, could its Hive (on EMR) still function the same way after the DB upgrade, as it did before? And are there any foreseeable issues for such an upgrade like this?

asked 7 months ago315 views
1 Answer
Accepted Answer


Yes, EMR on Hive functions as usual. Mostly there wont be any issue in the external RDS upgrade as far as I know. As long as the external store that are compatible with MariaDB JDBC as your metastore, it works fine. However, I strongly recommend you to first test in your lower environment and have the necessary backup plans like have the backup of database and EMR side configurations in place before make any changes.

Post upgrade, Metastore might not be auto-initialized. If that is the case, use the hive schema tool to perform them. - https://cwiki.apache.org/confluence/display/Hive/Hive+Schema+Tool

This doc might help to refer the steps - https://aws.amazon.com/blogs/big-data/migrate-and-deploy-your-apache-hive-metastore-on-amazon-emr/

Prechecks before mysql upgrade - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.MySQL.html#USER_UpgradeDBInstance.MySQL.57to80Prechecks

answered 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions