Cannot drop iceberg table by SparkSQL with AWS Glue Catalog

0
  • EMR Version: 6.15.0
  • Spark Conf
    • "spark.sql.catalog.spark_catalog": "org.apache.iceberg.spark.SparkSessionCatalog"
    • "spark.sql.catalog.spark_catalog.catalog-impl": "org.apache.iceberg.aws.glue.GlueCatalog"
    • "spark.sql.catalog.spark_catalog.io-impl": "org.apache.iceberg.aws.s3.S3FileIO"
    • "spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"

Create Table, write, read is OK. But I get the following error when I drop table:

spark.sql("drop table spark_catalog.$db.test_iceberg_0 purge")

pyspark.errors.exceptions.captured.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to fetch table test_iceberg_0. StorageDescriptor#InputFormat cannot be null for table: test_iceberg_0 (Service: null; Status Code: 0; Error Code: null; Request ID: null; Proxy: null)

duwan
질문됨 3달 전257회 조회
1개 답변
0

That error seems to indicate it is not using the Iceberg catalog extension, does it work if you SELECT the same table? Do you really need to use "purge"?, without that it shouldn't error (maybe that is not supported on Iceberg tables)

profile pictureAWS
전문가
답변함 3달 전
  • Thank you for your help.

    1. SELECT the same table is OK.
    2. I tried without purge and got the same error..

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인