Cannot drop iceberg table by SparkSQL with AWS Glue Catalog

0
  • EMR Version: 6.15.0
  • Spark Conf
    • "spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkSessionCatalog"
    • "spark.sql.catalog.spark_catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog"
    • "spark.sql.catalog.spark_catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO"
    • "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"

Create Table, write, read is OK. But I get the following error when I drop table:

spark.sql("drop table spark_catalog.$db.test_iceberg_0 purge")

pyspark.errors.exceptions.captured.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table test_iceberg_0. StorageDescriptor#InputFormat cannot be null for table: test_iceberg_0 (Service: null; Status Code: 0; Error Code: null; Request ID: null; Proxy: null)

duwan
質問済み 3ヶ月前256ビュー
1回答
0

That error seems to indicate it is not using the Iceberg catalog extension, does it work if you SELECT the same table? Do you really need to use "purge"?, without that it shouldn't error (maybe that is not supported on Iceberg tables)

profile pictureAWS
エキスパート
回答済み 3ヶ月前
  • Thank you for your help.

    1. SELECT the same table is OK.
    2. I tried without purge and got the same error..

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン

関連するコンテンツ