2 Respostas
- Mais recentes
- Mais votos
- Mais comentários
1
That sounds like a detect, when using the same Delta sink on a regular non streaming job, the code looks like the second one you describe. Will open a ticket to the streaming team.
0
No, it is correct. In the same Visual Job add a parent data node (Kafka in my case). Once this is in place you will see that the script changes to something like:
s3 = glueContext.write_dynamic_frame.from_options(
frame=df,
connection_type="s3",
format="delta",
connection_options={
"path": "s3://...",
"partitionKeys": [],
},
format_options={"compression": "snappy"},
transformation_ctx="s3",
)
I made the same test you did and is how discovered that this version works:
S3bucket_node3_df.write.format("delta").options(**additional_options).mode("append").save()
respondido há 9 meses
The streaming is treating delta (as well as hudi and iceberg) like one of the basic formats; anyway, the bug is reported
Conteúdo relevante
- AWS OFICIALAtualizada há um ano
- AWS OFICIALAtualizada há 2 anos
I've run into a very similar error trying to write to an iceberg table from a Glue streaming job.