is it possible to converta spark dataframe to dynamic frame and then using bookmark feature on the s3 folder used to read data in spark frame

0
df = spark.read.parquet("s3://folder/")
df = df.withColumn('filename', input_file_name())
AmazonS3_node1697616892615 = DynamicFrame.fromDF(df, glueContext, "s3sparkread")

if this is the code will the folder be bookmarked ? i want to read only new data coming to the folder.

i need to read in spark data frame as i am facing this issue [https://repost.aws/questions/QULSA0zge2QN-_2c_BqIQ_XA/in-aws-glue-not-able-to-add-the-source-file-name-in-the-dynamic-frame]( repost link)

1 Antwort
0

No, that will just transform the data already read but won't keep track of files.
Bookmarks require a DynamicFrame and a transaction_ctx specified.

profile pictureAWS
EXPERTE
beantwortet vor 4 Monaten
  • what can i change in this code to do bookmarking , i cannot read the data directly in to dynamic frame as i need the s3 file name in one of the columns.

    Do i need to modify this line to have some thing more

    AmazonS3_node1697616892615 = DynamicFrame.fromDF(df, glueContext, "s3sparkread")

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen