AWS Glue Interactive Notebooks: default df.saveAsTable() behavior overwrites entire table instead of dynamically based on dataframe values

I tried both of the following methods and get the same issue in a Glue notebook while simultaneously reading from the table in Athena for testing purposes. The table definition should not be completely recreated and partitions re-added when "overwriting" a new partition. In my test, the data is only a new partition value.

The biggest issue with this behavior is that, when a job is long-running, fails when writing data, or is killed, the table may not be able to be read until it is manually recreated.

I also wonder if this type of behavior is because I am using Glue as the data catalog.

Table <table name> does not exist
# then table exists with no data showing up
# then previous and new data shows up

    (
        df.write.option("compression", "snappy")
        .saveAsTable(
            name=target_db_table,
            mode="overwrite",
            format="parquet",
            partitionBy=partition_col_names,
            partitionOverwriteMode="dynamic",
        )
    )

    (
        df.write.option("compression", "snappy")
        .option("partitionOverwriteMode", "dynamic")
        .saveAsTable(
            name=target_db_table,
            mode="overwrite",
            format="parquet",
            partitionBy=partition_col_names,
        )
    )

Update: this is not ideal, but I was able to get new partitions to append dynamically and not drop/recreate the table by using the following:

def table_exists(
    spark_session: SparkSession,
    db_table: str,
) -> bool:
    try:
        spark_session.sql(f"desc {db_table}")
        exists = True
    except AnalysisException:
        exists = False

    return exists

    # If table does not exist, provide S3 path when saving to create the table dynamically as an external table
    if not table_exists(spark_session=spark_session, db_table=target_db_table):
        (
            df.write.option("compression", write_compression)
            .option("path", target_s3_path)
            .saveAsTable(
                name=target_db_table,
                mode=write_mode,
                format=write_format,
                partitionBy=partition_col_names,
            )
        )
    # Otherwise, dynamically overwrite existing partition value combinations and append new partitions
    else:
        (
            df.select(spark_session.table(target_db_table).columns).write.insertInto(
                tableName=target_db_table,
                overwrite=True,
            )
        )

I don't think ".option("partitionOverwriteMode", "dynamic")" is doing anyway, without specifying the full property name