Is it not possible to create a partioned iceberg table with AWS Glue createTable?

0

I'm calling glue to create an iceberg table, but neither PartitionIndexes or TableInput.PartitionKeys is accepted by Iceberg tables.

await glue.createTable({
  DatabaseName: 'test',
  OpenTableFormatInput: {
    IcebergInput: {
      MetadataOperation: 'CREATE',
    },
  },
  TableInput: {
    Name: 'test',
    StorageDescriptor: {
      Columns: [
        { Name: 'column_a', Type: 'date' },
        { Name: 'column_b', Type: 'string' },
      ],
      Location: 's3://test/test',
    },
  },
}).promise();

Is there something equivalent to using PARTITION BY in athena?

asked 7 months ago181 views
1 Answer
0

You shouldn't create Iceberg tables manually, configure the Iceberg library to use the Glue catalog and register a reference when writing the data.
https://iceberg.apache.org/docs/latest/aws/#spark
Iceberg doesn't use standard table partitions, the partitions in Iceberg are handled internally by the metadata (so they are dynamic).

profile pictureAWS
EXPERT
answered 6 months ago
  • There's literally an API to create iceberg tables with glue, and I'm not using spark at all here so I'm not sure how any of this is really relevant.

  • I believe that API registers an existing table on the catalog but you still need an engine to create the actual table like Spark or Athena, it's easier if you do both things at the same time (create the table on s3 and register it on the catalog)

  • You don't need to do anything when creating the table via the glue API. You can query the (empty) table, or insert into it like normal. It's a fully functional iceberg table, it's just not clear how to do the call to set it up with partitions like PARTITION BY in athena, which is all the question is about really.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions