Creating Partitions in Glue Tables

0

I have the following use case: Log audit events that come across a an Event Bus

To accomplish this I am doing the following:

  • Creating an S3 bucket to hold the events -- the events are stored as follows: eventsource\eventname\events where eventsource is the s3 bucket
  • Creating a Glue crawler to crawl eventsource\eventname to determine the schema of events and create the appropriate table
  • I would like the table partitioned by eventname

Everything is being created in code (javascript) using the SDK. I can't seem to force the creation of the partitions. When I try to create the partition using the following code I receive the error

InvalidInputException: The number of partition keys do not match the number of partition values
const createPartition = async (db, table, partition) => {
    let params;
    try{

        let StorageDescriptor;
        ({ Table:{StorageDescriptor} } = await glue.getTable({
            DatabaseName: db,
            Name: table,
        }).promise());
        let location = `${StorageDescriptor.Location}${partition}/`;
        params = {
            DatabaseName: db,
            TableName: table,
            PartitionInput: {
                StorageDescriptor: {
                    ...StorageDescriptor,
                    Location: location
                },
                [ partition ]
            }
        };

        let result = await glue.createPartition(params).promise();
        console.log("Create Partition Success:", result);
        return true;

    } catch(err) {
        console.log("Create Partition Error:", params, err);
        throw err;
    }
};

I don't see in the documentation of Glue or the createPartition method anything about defining partition keys.

Any feedback is appreciated.

asked 2 years ago852 views
1 Answer
0

You can specify the partition columns when creating Glue table, see API doc for detail.

AWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions