Glue Tables - Tag Parameters in PartitionKeys

0

Hello,

I'm creating a table following the documentation boto3, but when I run the create_table command, I get the error An error occurred (InvalidInputException) when calling the CreateTable operation: Parameters not supported for partition columns.

In the AWS documentation , the parameters tag inside the partitionKeys is supported but I tried to create the tag in the aws console and got the same error.

I used a code similar to this using the boto3 API

import boto3

glue = boto3.client('glue', region_name='us-east-1')

table_input = {
    'Name': 'dummyTable',
    'Description': 'Test parameters on partitionKeys',
    'TableType': 'EXTERNAL_TABLE',
    'StorageDescriptor': {
        'Columns': [
                      {
                        "Name": "id",
                        "Type": "int",
                        "Parameters": {
                          "type": "int",
                          "index": "true"
                        }
                      },
                      {
                        "Name": "name",
                        "Type": "string"
                      },
                      {
                        "Name": "age",
                        "Type": "bigint"
                      },
                      {
                        "Name": "city",
                        "Type": "string"
                      }
                    ],
        'Location': "s3://my-bucket/dummytable",
        "Parameters": {
            "classification": "parquet",
            "typeOfData": "file"
        },
        'InputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat',
        'OutputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat',
        'SerdeInfo': {'SerializationLibrary': 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
        }
    },
    'PartitionKeys': [
            {
                'Name': 'active',
                'Type': 'int',
                'Comment': 'Test Partition',
                'Parameters': {
                    'index': 'True'
                }
            },
        ]
}

createTable = glue.create_table(
        DatabaseName='sample',
        TableInput=table_input)
profile picture
marcus
質問済み 1年前511ビュー
3回答
0

Hello,

I have tested this and can confirm that adding Parameters under PartitionKeys is throwing below error.

Error: botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the CreateTable operation: Parameters not supported for partition columns

I am going to further test this and will update soon.

Thanks Nihar

AWS
回答済み 1年前
0

Hi,

This seems to be a documentation issue. The PartitionKeys object is an array of Column objects, which contain the Parameters property. This property can be set in other Column objects, but not when that Column represents a partition column in a table, according to our code. It looks like our documentation was made with the knowledge that columns can have parameters - but not knowing this limitation. We support parameters on normal table columns for the Lake Formation Michigan use cases - so we are going to open discussion as to whether this capability should be opened everywhere. In the meantime, we have put in a request to change the documentation to reflect what is currently possible.

Thanks Nihar


Note : Please upvote/accept my answer if it helped you.

AWS
回答済み 1年前
0

Ninar, do you think we can use this parameter in future updates?

profile picture
marcus
回答済み 1年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ