使用 AWS re:Post 即表示您同意 AWS re:Post 使用條款
AWS re:Postre:Post

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to build AWSGlueClient: com.amazonaws.SdkClientExcep

0
AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Unable to build AWSGlueClient: com.amazonaws.SdkClientException: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.)

Getting this error when defining spark views in AWS Glue local development using Docker image. Below is a function that I have defined and trying to call in the PySpark code ,which results in this error.

def incrementalTransform(glueContext, dfc) -> DynamicFrameCollection:
    input = dfc.select(list(dfc.keys())[0]).toDF()
    target = dfc.select(list(dfc.keys())[1]).toDF()
    targetWithColumns = (
        target.withColumn("rownum", lit(0))
        .withColumn("partition_year", year("create_time"))
        .withColumn("partition_month", month("create_time"))
        .withColumn("partition_day", dayofmonth("create_time"))
    )

    inputView = input.createOrReplaceTempView("inputView")
    targetView = targetWithColumns.createOrReplaceTempView("outputView")

    r1 = spark.sql("(select * from inputView)")
    r2 = spark.sql("(select * from targetView)")
    result = spark.sql("(select * from inputView) UNION All (select * from targetView)")
    r1.show()
    r2.show()
    result.show()
  • lprevost
    1 年前

    I'm running into same error with profile passed and aws credentials passed. AWS credentials include:

    [profile dev] region = us-east-1 output = json powershell script: $JUPYTER_WORKSPACE_LOCATION=$pwd -replace "\", "/" $AWS_PROFILE = "dev" docker run -it -v ~/.aws:/home/glue_user/.aws -v ${JUPYTER_WORKSPACE_LOCATION}:/home/glue_user/workspace/jupyter_workspace/ -e DISABLE_SSL=true -e AWS_PROFILE=$AWS_PROFILE --rm -p 4040:4040 -p 18080:18080 -p 8998:8998 -p 8888:8888 --name glue_jupyter_lab amazon/aws-glue-libs:glue_libs_4.0.0_image_01 /home/glue_user/jupyter/jupyter_start.sh

主題
分析
標籤
AWS Glue
語言
English
KoustavC
已提問 3 年前檢視次數 2390 次
1 個回答
0

Hello,

Looking at the error Unable to find a region via the region provider chain it appears that your code/script is unable to retrieve the AWS region specification from the available configuration files or environment within the docker container.

As per this article , the below command is used while starting the docker container

docker run -it -v ~/.aws:/root/.aws:ro --name glue_jupyter amazon/aws-glue-libs:glue_libs_1.0.0_image_01 bash

This will mount the .aws folder from your local filesystem onto the docker container. Usually, this .aws folder has the region specification set in a .config file. A sample config file looks like below

cat ~/.aws/config
[default]
region = us-east-1

You can find more about the aws configuration files here

AWS
支援工程師
Chiranjeevi_N
已回答 3 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋，同時有助於提問者的專業成長。

回答問題指南

相關內容