AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to build AWSGlueClient: com.amazonaws.SdkClientExcep

Question

```
AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to build AWSGlueClient: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.)
```

Getting this error when defining spark views in AWS Glue local development using Docker image. Below is a function that I have defined and trying to call in the PySpark code ,which results in this error.

```
def incrementalTransform(glueContext, dfc) -> DynamicFrameCollection:
    input = dfc.select(list(dfc.keys())[0]).toDF()
    target = dfc.select(list(dfc.keys())[1]).toDF()
    targetWithColumns = (
        target.withColumn("rownum", lit(0))
        .withColumn("partition_year", year("create_time"))
        .withColumn("partition_month", month("create_time"))
        .withColumn("partition_day", dayofmonth("create_time"))
    )

inputView = input.createOrReplaceTempView("inputView")
    targetView = targetWithColumns.createOrReplaceTempView("outputView")

r1 = spark.sql("(select * from inputView)")
    r2 = spark.sql("(select * from targetView)")
    result = spark.sql("(select * from inputView) UNION All (select * from targetView)")
    r1.show()
    r2.show()
    result.show()
```

Answer

Hello,

Looking at the error `Unable to find a region via the region provider chain` it appears that your code/script is unable to retrieve the AWS region specification from the available configuration files or environment within the docker container.

As per this [article](https://aws.amazon.com/blogs/big-data/developing-aws-glue-etl-jobs-locally-using-a-container/) , the below command is used while starting the docker container

```
docker run -it -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_1.0.0_image_01 bash
```
This will mount the **.aws** folder from your local filesystem onto the docker container. Usually, this **.aws** folder has the region specification set in a **.config** file. A sample config file looks like below

```
cat ~/.aws/config
[default]
region = us-east-1
```

You can find more about the aws configuration files [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html#cli-configure-files-where)

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to build AWSGlueClient: com.amazonaws.SdkClientExcep

相关内容