Create a glue table with type delta using AWS CDK

You can create tables using either the L1 constructor or the L2 constructor, whichever you prefer.
If you are familiar with CloudFormation, the L1 constructor may be easier to create.
https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_glue_alpha/Table.html
https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_glue/CfnTable.html

I'm a little late on this, but I was able to create a Glue table using AWS CDK (level 1 constructs - aws_cdk/aws_glue using Python). I first generated the table using the CREATE EXTERNAL TABLE Athena DDL command. I examined the generated table, then examined its properties to reverse engineer what was needed in the CDK. To make sure this worked, I also deleted the table generated by the CREATE EXTERNAL TABLE DDL, then ran the CDK separately.

I had previously created my Glue database using the following CDK:

my_database: CfnDatabase = CfnDatabase(
            self,
            "my_database",
            catalog_id=self.account,
            database_input={
                "name": "my_database"
            }
        )

Here's my CDK for the table definition:

CfnTable(
    self,
    "my_table",
    catalog_id=self.account,
    database_name=my_database.database_input.name,
    table_input={
        "name": "my_table",
        "tableType": "EXTERNAL_TABLE",
        "parameters": {
            # "delta.lastUpdateVersion": "2",
            # "delta.lastCommitTimestamp": "1716930161932",
            "EXTERNAL": "TRUE",
            # "spark.sql.sources.schema.part.0": "{\"type\":\"struct\",\"fields\":" + json.dumps(
            #     my_data_schema) + "}",
            "spark.sql.sources.partitionProvider": "catalog",
            "spark.sql.sources.provider": "delta",
            # "spark.sql.sources.schema.numParts": "1",
            "table_type": "delta"
        },
        "partitionKeys": [{"name": "my_partition_field_column", "type": "string"}],
        "storageDescriptor": {
            "columns": my_data_schema,
            "location": "s3://mybucket/path-to-delta",
            "inputFormat": "org.apache.hadoop.mapred.SequenceFileInputFormat",
            "outputFormat": "org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat",
            "serdeInfo": {
                "name": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",
                "serializationLibrary": "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe",
                "parameters": {
                    "path": "s3://mybucket/path-to-delta",
                    "serialization.format": "1"
                }
            }
        }
    }
)

The major annoyance is I had to include my schema as Glue could not automatically determine my schema (unlike the CREATE EXTERNAL TABLE) command. So the schema I have in my_data_schema looks something like:

my_data_schema: list[dict[str, Union[str, dict]]] = [
    {"name": "field_1", "type": "decimal(10,2)", "nullable": True, "metadata": {}},
    {"name": "field_2", "type": "string", "nullable": True, "metadata": {}},
    ...
]

Obviously, it would be nice if Glue can just determine the schema from the Delta table metadata, but the Athena queries failed with errors if I did not include the columns property under the storageDescriptor property.

I was able to comment out/delete fields like delta.lastUpdateVersion, delta.lastCommitTimestamp, spark.sql.sources.schema.part.0, and spark.sql.sources.schema.numParts as they didn't seem to be needed for my Athena queries to work. I didn't test out every single field so there may be a few others that could be removed without causing issues.

I'm still hoping I don't have to explicitly define the schema in the CDK code. If anyone can comment on how I can get all this to work without defining the schema in the CDK, I would really appreciate it!