How to read the schema only from Glue Data Catalog
0
Hi Team,
I am using below code and its giving me the columns from the data, but my expectation to get the columns from Glue Data Catalog.
glueContext.create_dynamic_frame.from_catalog(database = "xyz", table_name = "abc")
Step 1 : Kept 5 columns in a file in S3 and ran the crawler. Glue Data Catalog captured all 5 columns as schema
Step 2 : Executed a Glue Pyspark script job with glueContext.create_dynamic_frame.from_catalog and loaded data Iceberg table with merge statement.
Step 3 : Removed one column from step1 and Kept 4 columns in a file in S3 as delta file(cdc) and ran the crawler. Glue Data Catalog captured all 4 columns
Step 4 : Executed the same Pyspark script job with glueContext.create_dynamic_frame.from_catalog and the job failed with an error : AnalysisException: cannot resolve colunm_name in MERGE command.