I have an IAM role for my bucket that has the following permissions:
"s3:PutObject",
"s3:GetObject",
"iam:PassRole",
"kms:Decrypt",
"kms:Encrypt",
"kms:GenerateDataKey",
"s3:ListBucket",
"s3:DeleteObject"
I've set the configuration in the following way within my AWS Glue Notebook:
%%configure
("--conf":"spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension --conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore",
"--datalake-formats": "delta"
)
I then perform my transformations and attempt to save the spark dataframe using the following code:
additional_options = {"path": "s3://bucket_name/prefix/path/for/file/", "write.parquet.compression-codec": "snappy"}
df.write.format("delta").options(**additional_options).mode("append").save()
which results in the following error:
IllegalArgumentException: Wrong FS: s3:/prefix/path/for/file/_delta_log, expected: s3://bucket_name
For some reason, the bucket_name seems to disappear here. The bucket name does have "." in it. Could this be a reason for the error by any chance? I'm fairly new to pyspark and glue so I don't know if I have configured something wrong in my notebook/permissions or if I'm using pyspark incorrectly here.