is it possible to converta spark dataframe to dynamic frame and then using bookmark feature on the s3 folder used to read data in spark frame

df = spark.read.parquet("s3://folder/")
df = df.withColumn('filename', input_file_name())
AmazonS3_node1697616892615 = DynamicFrame.fromDF(df, glueContext, "s3sparkread")

if this is the code will the folder be bookmarked ? i want to read only new data coming to the folder.

i need to read in spark data frame as i am facing this issue [https://repost.aws/questions/QULSA0zge2QN-_2c_BqIQ_XA/in-aws-glue-not-able-to-add-the-source-file-name-in-the-dynamic-frame]( repost link)

주제

스토리지 분석 데이터베이스

태그

아마존 심플 스토리지 서비스 AWS Glue 데이터 추출, 변환 및 로드

언어

English

질문됨 4달 전317회 조회

1개 답변

최신
최다 투표
가장 많은 댓글

이 답변이 도움이 되었나요?커뮤니티가 여러분의 지식을 활용할 수 있도록 정답을 찬성하세요.

No, that will just transform the data already read but won't keep track of files.
Bookmarks require a DynamicFrame and a transaction_ctx specified.

전문가

Gonzalo Herreros

답변함 4달 전

KG
4달 전
what can i change in this code to do bookmarking , i cannot read the data directly in to dynamic frame as i need the s3 file name in one of the columns.

Do i need to modify this line to have some thing more

AmazonS3_node1697616892615 = DynamicFrame.fromDF(df, glueContext, "s3sparkread")

is it possible to converta spark dataframe to dynamic frame and then using bookmark feature on the s3 folder used to read data in spark frame

관련 콘텐츠