Creating Partitions in Glue Tables

0

I have the following use case: Log audit events that come across a an Event Bus

To accomplish this I am doing the following:

  • Creating an S3 bucket to hold the events -- the events are stored as follows: eventsource\eventname\events where eventsource is the s3 bucket
  • Creating a Glue crawler to crawl eventsource\eventname to determine the schema of events and create the appropriate table
  • I would like the table partitioned by eventname

Everything is being created in code (javascript) using the SDK. I can't seem to force the creation of the partitions. When I try to create the partition using the following code I receive the error

InvalidInputException: The number of partition keys do not match the number of partition values
const createPartition = async (db, table, partition) => {
    let params;
    try{

        let StorageDescriptor;
        ({ Table:{StorageDescriptor} } = await glue.getTable({
            DatabaseName: db,
            Name: table,
        }).promise());
        let location = `${StorageDescriptor.Location}${partition}/`;
        params = {
            DatabaseName: db,
            TableName: table,
            PartitionInput: {
                StorageDescriptor: {
                    ...StorageDescriptor,
                    Location: location
                },
                [ partition ]
            }
        };

        let result = await glue.createPartition(params).promise();
        console.log("Create Partition Success:", result);
        return true;

    } catch(err) {
        console.log("Create Partition Error:", params, err);
        throw err;
    }
};

I don't see in the documentation of Glue or the createPartition method anything about defining partition keys.

Any feedback is appreciated.

已提問 2 年前檢視次數 864 次
1 個回答
0

You can specify the partition columns when creating Glue table, see API doc for detail.

AWS
已回答 2 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南