In AWS Glue not able to add the source file name in the dynamic frame

I tried to add through 2 ways

    newdf = newdf.withColumn('filename2', input_file_name())

also tried

AmazonS3_node = glueContext.create_dynamic_frame.from_options(
    connection_type="s3",
    format="parquet",
    connection_options={"paths": [ "s3://bucket/"], "recurse": True},
    transformation_ctx="AmazonS3_node",
    format_options={
        "attachFilename": "filename"
    }
)

both of them dont work also in 2nd case the column is not created

Please help with this

主题

存储分析

标签

亚马逊简单存储服务 (S3)AWS Glue

语言

English

已提问 4 个月前447 查看次数

1 回答

最新
投票最多
评论最多

这些答案有用吗？为正确答案投票，以帮助社区从您的知识中受益。

I believe the attachFilename doesn't work for all formats.
Try using format="glueparquet", otherwise you could use the first example but reading directly with DataFrame (and then convert to DynamicFrame if you want to):

newdf = spark.read.parquet(s3://bucket/)
newdf = newdf.withColumn('filename2', input_file_name())

专家

Gonzalo Herreros

已回答 4 个月前

KG
4 个月前
But if i read through spark directly, would i be still able to use bookmark ?
Yotam Levy
3 个月前
In the documentation, it says that attachFilename can be used with any format, I've tested it with CSV and it works. Do you have an idea why for Parquet it's different?

This is the link to the section talking about attachFIlename option: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format.html#aws-glue-programming-etl-format-shared-reference

In AWS Glue not able to add the source file name in the dynamic frame

相关内容