By using AWS re:Post, you agree to the Terms of Use
/AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to build AWSGlueClient: com.amazonaws.SdkClientExcep/

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to build AWSGlueClient: com.amazonaws.SdkClientExcep

0
AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to build AWSGlueClient: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.)

Getting this error when defining spark views in AWS Glue local development using Docker image. Below is a function that I have defined and trying to call in the PySpark code ,which results in this error.

def incrementalTransform(glueContext, dfc) -> DynamicFrameCollection:
    input = dfc.select(list(dfc.keys())[0]).toDF()
    target = dfc.select(list(dfc.keys())[1]).toDF()
    targetWithColumns = (
        target.withColumn("rownum", lit(0))
        .withColumn("partition_year", year("create_time"))
        .withColumn("partition_month", month("create_time"))
        .withColumn("partition_day", dayofmonth("create_time"))
    )

    inputView = input.createOrReplaceTempView("inputView")
    targetView = targetWithColumns.createOrReplaceTempView("outputView")

    r1 = spark.sql("(select * from inputView)")
    r2 = spark.sql("(select * from targetView)")
    result = spark.sql("(select * from inputView) UNION All (select * from targetView)")
    r1.show()
    r2.show()
    result.show()
asked a month ago14 views
1 Answers
0

Hello,

Looking at the error Unable to find a region via the region provider chain it appears that your code/script is unable to retrieve the AWS region specification from the available configuration files or environment within the docker container.

As per this article , the below command is used while starting the docker container

docker run -it -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_1.0.0_image_01 bash

This will mount the .aws folder from your local filesystem onto the docker container. Usually, this .aws folder has the region specification set in a .config file. A sample config file looks like below

cat ~/.aws/config
[default]
region = us-east-1

You can find more about the aws configuration files here

answered 22 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions