Glue Tables - Tag Parameters in PartitionKeys

0

Hello,

I'm creating a table following the documentation boto3, but when I run the create_table command, I get the error An error occurred (InvalidInputException) when calling the CreateTable operation: Parameters not supported for partition columns.

In the AWS documentation , the parameters tag inside the partitionKeys is supported but I tried to create the tag in the aws console and got the same error.

I used a code similar to this using the boto3 API

import boto3

glue = boto3.client('glue', region_name='us-east-1')

table_input = {
    'Name': 'dummyTable',
    'Description': 'Test parameters on partitionKeys',
    'TableType': 'EXTERNAL_TABLE',
    'StorageDescriptor': {
        'Columns': [
                      {
                        "Name": "id",
                        "Type": "int",
                        "Parameters": {
                          "type": "int",
                          "index": "true"
                        }
                      },
                      {
                        "Name": "name",
                        "Type": "string"
                      },
                      {
                        "Name": "age",
                        "Type": "bigint"
                      },
                      {
                        "Name": "city",
                        "Type": "string"
                      }
                    ],
        'Location': "s3://my-bucket/dummytable",
        "Parameters": {
            "classification": "parquet",
            "typeOfData": "file"
        },
        'InputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat',
        'OutputFormat': 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat',
        'SerdeInfo': {'SerializationLibrary': 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
        }
    },
    'PartitionKeys': [
            {
                'Name': 'active',
                'Type': 'int',
                'Comment': 'Test Partition',
                'Parameters': {
                    'index': 'True'
                }
            },
        ]
}

createTable = glue.create_table(
        DatabaseName='sample',
        TableInput=table_input)
profile picture
marcus
질문됨 일 년 전511회 조회
3개 답변
0

Hello,

I have tested this and can confirm that adding Parameters under PartitionKeys is throwing below error.

Error: botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the CreateTable operation: Parameters not supported for partition columns

I am going to further test this and will update soon.

Thanks Nihar

AWS
답변함 일 년 전
0

Hi,

This seems to be a documentation issue. The PartitionKeys object is an array of Column objects, which contain the Parameters property. This property can be set in other Column objects, but not when that Column represents a partition column in a table, according to our code. It looks like our documentation was made with the knowledge that columns can have parameters - but not knowing this limitation. We support parameters on normal table columns for the Lake Formation Michigan use cases - so we are going to open discussion as to whether this capability should be opened everywhere. In the meantime, we have put in a request to change the documentation to reflect what is currently possible.

Thanks Nihar


Note : Please upvote/accept my answer if it helped you.

AWS
답변함 일 년 전
0

Ninar, do you think we can use this parameter in future updates?

profile picture
marcus
답변함 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인