Skip to content

Delta tables error on AWS glue notebook

0

I have an IAM role for my bucket that has the following permissions: "s3:PutObject", "s3:GetObject", "iam:PassRole", "kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey", "s3:ListBucket", "s3:DeleteObject"

I've set the configuration in the following way within my AWS Glue Notebook:

%%configure
("--conf":"spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore",
"--datalake-formats": "delta"
)

I then perform my transformations and attempt to save the spark dataframe using the following code:

additional_options = {"path": "s3://bucket_name/prefix/path/for/file/", "write.parquet.compression-codec": "snappy"}
df.write.format("delta").options(**additional_options).mode("append").save()

which results in the following error:

IllegalArgumentException: Wrong FS: s3:/prefix/path/for/file/_delta_log, expected: s3://bucket_name

For some reason, the bucket_name seems to disappear here. The bucket name does have "." in it. Could this be a reason for the error by any chance? I'm fairly new to pyspark and glue so I don't know if I have configured something wrong in my notebook/permissions or if I'm using pyspark incorrectly here.

1 Answer
0
Accepted Answer

I was able to fix this error by adding an extra configuration argument. Specifically, --conf spark.hadoop.fs.defaultFS=s3://bucket_name

answered 7 months ago

AWS
EXPERT

reviewed 7 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.