We are working on Glue ETL job with visual mode and have multiple S3 (Glue Data Catalog type) source nodes that we want to use Pushdown Predicate to implement daily updating task.
- The Glue table format is COW Apache Hudi, created natively by Glue ETL job
- The Glue table successfully created with Partition registered in Hudi, Hive and Glue level
- The table's partition is mixed of string and integer type
We have created the predicate string and put it to S3 source node, then SAVE, Refresh the webpage and found that the predicate option did not intact.
Inspecting the packet from AWS console to Glue Rest API, found that the console ignored the configuration and did not POST/save it to ETL Python script.
With this limitation, or bug, we have to clone the job to script-only mode and add the option manually.
How can we save the configuration with the original visual mode?
AWS 官方已更新 3 年前
cannot reproduce that, maybe there is something in the predicate that breaks the saving, can you share it?