Creating Partitions in Glue Tables

0

I have the following use case: Log audit events that come across a an Event Bus

To accomplish this I am doing the following:

  • Creating an S3 bucket to hold the events -- the events are stored as follows: eventsource\eventname\events where eventsource is the s3 bucket
  • Creating a Glue crawler to crawl eventsource\eventname to determine the schema of events and create the appropriate table
  • I would like the table partitioned by eventname

Everything is being created in code (javascript) using the SDK. I can't seem to force the creation of the partitions. When I try to create the partition using the following code I receive the error

InvalidInputException: The number of partition keys do not match the number of partition values
const createPartition = async (db, table, partition) => {
    let params;
    try{

        let StorageDescriptor;
        ({ Table:{StorageDescriptor} } = await glue.getTable({
            DatabaseName: db,
            Name: table,
        }).promise());
        let location = `${StorageDescriptor.Location}${partition}/`;
        params = {
            DatabaseName: db,
            TableName: table,
            PartitionInput: {
                StorageDescriptor: {
                    ...StorageDescriptor,
                    Location: location
                },
                [ partition ]
            }
        };

        let result = await glue.createPartition(params).promise();
        console.log("Create Partition Success:", result);
        return true;

    } catch(err) {
        console.log("Create Partition Error:", params, err);
        throw err;
    }
};

I don't see in the documentation of Glue or the createPartition method anything about defining partition keys.

Any feedback is appreciated.

gefragt vor 2 Jahren864 Aufrufe
1 Antwort
0

You can specify the partition columns when creating Glue table, see API doc for detail.

AWS
beantwortet vor 2 Jahren

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen