Cannot save Pushdown Predicate for S3 source's Glue Table in Glue Job visual mode

0

We are working on Glue ETL job with visual mode and have multiple S3 (Glue Data Catalog type) source nodes that we want to use Pushdown Predicate to implement daily updating task.

  • The Glue table format is COW Apache Hudi, created natively by Glue ETL job
  • The Glue table successfully created with Partition registered in Hudi, Hive and Glue level
  • The table's partition is mixed of string and integer type

We have created the predicate string and put it to S3 source node, then SAVE, Refresh the webpage and found that the predicate option did not intact. Inspecting the packet from AWS console to Glue Rest API, found that the console ignored the configuration and did not POST/save it to ETL Python script.

With this limitation, or bug, we have to clone the job to script-only mode and add the option manually. How can we save the configuration with the original visual mode?

  • cannot reproduce that, maybe there is something in the predicate that breaks the saving, can you share it?

asked 7 months ago160 views
1 Answer
0
Accepted Answer

As of today, the issue has been fixed and we can modified pushdown_predicate from the Glue interface. Checking Spark UI log found that the filter is intacted.

answered 6 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions