Glue script job error spark_catalog requires a single-part namespace, but got [glue_catalog, foo]

Question

I am trying to query an iceberg table via the glue data catalog. This works fine in my visual etl job, but when I try to do it in a script, it's throwing an error. It's likely due to some sort of settings, but I haven't been able to determine what. 
The code is
```
glueContext.create_data_frame.from_catalog(
        database="foo",
        table_name="bar",
    )
```
and the error is `AnalysisException: spark_catalog requires a single-part namespace, but got [glue_catalog, foo]`
The error itself seems to be something that the `from_catalog` method is doing, and I can't figure out what the root cause is. 
Any suggestions?

Answer

In the visual job, the Iceberg catalog is configured for you, in your script job, you have to do it yourself, otherwise it won't recognize the "glue_catalog" (which is really the Iceberg catalog backed by Glue).   
See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html#aws-glue-programming-etl-format-iceberg-enable

Glue script job error spark_catalog requires a single-part namespace, but got [glue_catalog, foo]

相关内容