2 réponses
- Le plus récent
- Le plus de votes
- La plupart des commentaires
1
That sounds like a detect, when using the same Delta sink on a regular non streaming job, the code looks like the second one you describe. Will open a ticket to the streaming team.
0
No, it is correct. In the same Visual Job add a parent data node (Kafka in my case). Once this is in place you will see that the script changes to something like:
s3 = glueContext.write_dynamic_frame.from_options(
frame=df,
connection_type="s3",
format="delta",
connection_options={
"path": "s3://...",
"partitionKeys": [],
},
format_options={"compression": "snappy"},
transformation_ctx="s3",
)
I made the same test you did and is how discovered that this version works:
S3bucket_node3_df.write.format("delta").options(**additional_options).mode("append").save()
répondu il y a 9 mois
The streaming is treating delta (as well as hudi and iceberg) like one of the basic formats; anyway, the bug is reported
Contenus pertinents
- demandé il y a 2 mois
- demandé il y a 7 mois
- demandé il y a un an
- AWS OFFICIELA mis à jour il y a 3 ans
- AWS OFFICIELA mis à jour il y a un an
- AWS OFFICIELA mis à jour il y a 2 ans
I've run into a very similar error trying to write to an iceberg table from a Glue streaming job.