Glue Job Error (An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema)
Hi Dears, Hope all is doing great!
I am working on a project to migrate data from Google Bigquery to Amazon S3, I just followed the project in AWS blogs ( https://aws.amazon.com/blogs/big-data/migrating-data-from-google-bigquery-to-amazon-s3-using-aws-glue-custom-connectors/) which is a great example.
But when I did all the needed steps I faced this issue: error was : An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema
Can you please guide me on how can I fix the problem?
Thanks for your answer in advance! Basem
the error seems to indicate that the step that initiated the write to the target got as input an empty DynamicFrame. Are you sure the table you are reading is populated?
You should try to look at the logs of the job.
In Glue Studio open the jobs then select the tab Runs or in the left pane select Monitoring then follow the instruction in this documentation page. Look for other errors, that may have happen before as connection time out or privilege issue that might have not interrupted the job but led to an empty input.
After you read the data, do you do any processing?
If there are no error and you can manage the code in PySpark, clone the job , open it in manual edit mode and then try to print the schema and show the content.
hope this helps,
Yes, It is populated I found the problem, I didn't define schema for input and output data sources. I set it up and the job succeeded
Thanks for your help!
An error occurred while calling o280.pyWriteDynamicFrameasked 4 years ago
Glue ETL PySpark Job Fails after Upgrade from Glue Version 2.0 to 3.0 error occurred while calling pyWriteDynamicFrame EOFException occurred while reading the port number from pyspark.daemon's stdoutasked 5 months ago
Glue ETL Job not working with error: o122.relationalize. com.amazonaws.services.glue.util.HadoopDataSourceJobBookmarkState cannot be cast to com.amazonaws.services.glue.util.RelationalizeJobBookmarkStasked 6 months ago
Error on Dashboard "An error occurred" and unable to create instance due to "Error loading AMI data".asked a month ago
Error Running Glue CrawlerAccepted Answerasked 3 years ago
Glue Job Error (An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema)Accepted Answerasked 3 months ago
I am unable to delete my AWS Amplify projectasked a month ago
An error occurred while calling o95.pyWriteDynamicFrame.asked 5 months ago
An error occurred while calling o128.pyWriteDynamicFrame. Record is incomplete.asked 3 months ago
AWS Glue gives error: "An error occurred while calling o285.save. File already exists:"asked 6 months ago