Glue Job Error (An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema)

Question

Hi Dears, 
Hope all is doing great!

I am working on a project to migrate data from Google Bigquery to Amazon S3, I just followed the project in AWS blogs ( https://aws.amazon.com/blogs/big-data/migrating-data-from-google-bigquery-to-amazon-s3-using-aws-glue-custom-connectors/) which is a great example.

But when I did all the needed steps I faced this issue: error was :
An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema

Can you please guide me on how can I fix the problem?

Thanks for your answer in advance!
Basem

Accepted Answer

Hi,

the error seems to indicate that the step that initiated the write to the target got as input an empty DynamicFrame.
Are you sure the table you are reading is populated?

You should try to look at the logs of the job.

In Glue Studio open the jobs then select the tab **Runs** or in the left pane select **Monitoring** then follow the instruction in this [documentation page](https://docs.aws.amazon.com/glue/latest/ug/monitoring-chapter.html#monitoring-job-run-logs). 
Look for other errors, that may have happen before as connection time out or privilege issue that might have not interrupted the job but led to an empty input.

After you read the data, do you do any processing?

If there are no error and you can manage the code in PySpark, [clone the job](https://docs.aws.amazon.com/glue/latest/ug/managing-jobs-chapter.html#clone-jobs) ,  open it in manual edit mode and then try to print the schema and show the content.

hope this helps,

Glue Job Error (An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema)

相关内容