EMR with Glue Data Catalog as Hive Metastore

Question

I am trying to have glue data catalog as the hive metastore, stood up the EMR(emr-6.15.0) with the following node classification config per AWS, and it always initialize a default glue catalog database there, is there a hive/EMR config for disabling that auto creation or use an alternative database in glue on start up?

Thank you

Installed applications
Hadoop 3.3.6, Hive 3.1.3, JupyterEnterpriseGateway 2.6.0, Livy 0.7.1, Spark 3.4.1

```
[
  {
    "Classification": "delta-defaults",
    "Properties": {
      "delta.enabled": "true"
    }
  },
  {
    "Classification": "spark-hive-site",
    "Properties": {
      "hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"
    }
  }
]
```

```
spark = SparkSession.builder \
.config("spark.jars", "/usr/share/aws/delta/lib/delta-core_2.12-2.4.0.jar") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.delta.catalog.DeltaCatalog") \
.config("hive.metastore.client.factory.class", "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory") \
.appName("test") \
.enableHiveSupport() \
.getOrCreate()
```

Accepted Answer

Hello,

As far as I know, you need default database in glue data catalog as a default database and deleting it might cause issues. In AWS Glue job’s catalog integration, AWS Glue always tries to check whether default database exists or not using AWS Glue GetDatabase API. When the DESCRIBE Lake Formation permission is not granted, or GetDatabase IAM permission is granted, then the job fails when verifying existence of the default database.

"default" database is the one that hive uses by default which can not be changed or set to other dbs. However,

To see the tables belonging to your "user_created_database": run,
> show tables from user_created_database;

Then, to run your queries, you can either specify the full table name using the notion database.table:
> select * from user_created_database.your_table

or you can change database (i.e., USE statement) and then run your queries:

> use user_created_database; 
> show tables;

EMR with Glue Data Catalog as Hive Metastore

Contenuto pertinente