Creating Partitions in Glue Tables

0

I have the following use case: Log audit events that come across a an Event Bus

To accomplish this I am doing the following:

  • Creating an S3 bucket to hold the events -- the events are stored as follows: eventsource\eventname\events where eventsource is the s3 bucket
  • Creating a Glue crawler to crawl eventsource\eventname to determine the schema of events and create the appropriate table
  • I would like the table partitioned by eventname

Everything is being created in code (javascript) using the SDK. I can't seem to force the creation of the partitions. When I try to create the partition using the following code I receive the error

InvalidInputException: The number of partition keys do not match the number of partition values
const createPartition = async (db, table, partition) => {
    let params;
    try{

        let StorageDescriptor;
        ({ Table:{StorageDescriptor} } = await glue.getTable({
            DatabaseName: db,
            Name: table,
        }).promise());
        let location = `${StorageDescriptor.Location}${partition}/`;
        params = {
            DatabaseName: db,
            TableName: table,
            PartitionInput: {
                StorageDescriptor: {
                    ...StorageDescriptor,
                    Location: location
                },
                [ partition ]
            }
        };

        let result = await glue.createPartition(params).promise();
        console.log("Create Partition Success:", result);
        return true;

    } catch(err) {
        console.log("Create Partition Error:", params, err);
        throw err;
    }
};

I don't see in the documentation of Glue or the createPartition method anything about defining partition keys.

Any feedback is appreciated.

질문됨 2년 전864회 조회
1개 답변
0

You can specify the partition columns when creating Glue table, see API doc for detail.

AWS
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠