Glue 4 Hudi support
I am trying to store a data stream from kafka using the hudi format. I am following this doc https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-hudi.html and I even tried to create a visual job. When the job attempts to write a batch I am always getting the following error:
py4j.protocol.Py4JJavaError: An error occurred while calling o372.pyWriteDynamicFrame. : java.lang.ClassNotFoundException: Failed to load format with name hudi
I am using Glue 4 but I get the same error with Glue 3. I have double-checked that the job parameters are exactly like those in the doc above.
Any suggestion?
- Language
- English
asked 3 years ago613 views
- Newest
- Most votes
- Most comments
Looks like the parameters are correct.
answered 3 years ago
That does look correct, you must have something that odd in your job (lots of people use Hudi like that). Do you have any connections? Maybe the issue is in the code call, have you tried using DataFrame?
Using a DataFrame works. If you can please report that the sample generated by the Visual tool that uses a DynamicFrame is buggy. Thanks
answered 3 years ago
You are right, it doesn't handle the s3 sink correctly for streaming visual jobs, reported.
I checked also the classpath and it looks ok to me.
- Yes the example uses a Kafka connection as data source
- The job is generated by the Visual tool
- I will try using a dataframe
answered 3 years ago
Relevant content
asked 2 years ago
asked 3 years ago
- AWS OFFICIALUpdated 2 years ago
Check the classpath on SparkUI, sounds the Hudi libraries are not added, which should happen if you did add the argument --datalake-formats=hudi