I have the following use case:
Log audit events that come across a an Event Bus
To accomplish this I am doing the following:
- Creating an S3 bucket to hold the events -- the events are stored as follows: eventsource\eventname\events where eventsource is the s3 bucket
- Creating a Glue crawler to crawl eventsource\eventname to determine the schema of events and create the appropriate table
- I would like the table partitioned by eventname
Everything is being created in code (javascript) using the SDK. I can't seem to force the creation of the partitions. When I try to create the partition using the following code I receive the error
InvalidInputException: The number of partition keys do not match the number of partition values
const createPartition = async (db, table, partition) => {
let params;
try{
let StorageDescriptor;
({ Table:{StorageDescriptor} } = await glue.getTable({
DatabaseName: db,
Name: table,
}).promise());
let location = `${StorageDescriptor.Location}${partition}/`;
params = {
DatabaseName: db,
TableName: table,
PartitionInput: {
StorageDescriptor: {
...StorageDescriptor,
Location: location
},
[ partition ]
}
};
let result = await glue.createPartition(params).promise();
console.log("Create Partition Success:", result);
return true;
} catch(err) {
console.log("Create Partition Error:", params, err);
throw err;
}
};
I don't see in the documentation of Glue or the createPartition method anything about defining partition keys.
Any feedback is appreciated.