Can I enable optimization in AWS Glue when I create an iceberg table instead of the using the console after the table is created?

0

Can I enable optimization in AWS Glue when I create an iceberg table? Is there an option like a tableProperty?

        df.writeTo(f"{table_name}") \
        .tableProperty("format-version", "2") \
        .using("iceberg") \
        .create()
asked 2 months ago92 views
1 Answer
0

Yes, you can enable optimization for Apache Iceberg tables in AWS Glue when you create the table, rather than using the console after table creation. When creating a new table, you have the option to choose Apache Iceberg as the table format and enable optimization features at that time.

However, it's important to note that by default, optimization features like compaction are disabled for new tables. To enable optimization during table creation, you would typically use the AWS Glue console, AWS CLI, or AWS API to set the appropriate properties.

While the code snippet you provided doesn't directly show how to enable optimization, you would need to use additional configuration options or API calls to enable optimization features when creating the table. The exact method would depend on the specific AWS Glue interface or SDK you're using.

For example, if you're using the AWS Glue API or CLI, you might include optimization configuration parameters when calling the create table operation. If you're using the AWS Glue console, you would have the option to enable optimization features during the table creation process.

It's worth noting that after creating the table, you can always enable or modify optimization settings using the AWS Glue console, CLI, or API operations if you didn't set them during table creation.
Sources
Enabling compaction optimizer - AWS Glue
AWS Glue Data Catalog now supports storage optimization of Apache Iceberg tables - AWS

profile picture
answered 2 months ago
profile picture
EXPERT
reviewed a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions