AWS Glue visual etl: Issues while overwriting files on s3

Root cause (by design behavior) In AWS Glue Visual ETL, the S3 target node defaults to append() semantics and does not currently expose a configurable write mode (overwrite) in the visual editor. This is intentional: Visual ETL is optimized for incremental pipelines and avoids destructive operations by default. As a result, every run appends data, which can lead to duplicates in downstream (silver/gold) layers.

Why this happens The generated Spark script is opinionated and managed by the visual abstraction. Even when using formats like Parquet or Iceberg, the visual S3 target still emits append logic unless custom code is introduced.

Recommended mitigation patterns 1. Use a custom Spark transform Insert a custom transform node and explicitly control the write logic, e.g. mode("overwrite"), partition overwrite, or conditional deletes. This is the most flexible and supported approach. 2. Pre-clean the S3 prefix Add a pre-action (Lambda or Glue job step) to delete the target S3 prefix before writing. This preserves Visual ETL while enforcing overwrite semantics. 3. Leverage table formats correctly For Iceberg/Hudi, manage overwrite or merge semantics at the table level (e.g., overwrite partitions, MERGE INTO) rather than relying on the visual S3 target defaults. 4. Architectural guidance Use Visual ETL primarily for bronze / incremental ingestion. For deterministic overwrite logic in silver layers, script-based Glue jobs or table-format–managed writes are a better fit.

Summary This is not a bug but a design limitation of Glue Visual ETL. When overwrite semantics are required, introducing a custom transform or managing writes at the table level is the recommended and supported pattern.