By using AWS re:Post, you agree to the Terms of Use
/Glue Job Error (An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema)/

Glue Job Error (An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema)

0

Hi Dears, Hope all is doing great!

I am working on a project to migrate data from Google Bigquery to Amazon S3, I just followed the project in AWS blogs ( https://aws.amazon.com/blogs/big-data/migrating-data-from-google-bigquery-to-amazon-s3-using-aws-glue-custom-connectors/) which is a great example.

But when I did all the needed steps I faced this issue: error was : An error occurred while calling o100.pyWriteDynamicFrame. Illegal empty schema

Can you please guide me on how can I fix the problem?

Thanks for your answer in advance! Basem

1 Answers
0
Accepted Answer

Hi,

the error seems to indicate that the step that initiated the write to the target got as input an empty DynamicFrame. Are you sure the table you are reading is populated?

You should try to look at the logs of the job.

In Glue Studio open the jobs then select the tab Runs or in the left pane select Monitoring then follow the instruction in this documentation page. Look for other errors, that may have happen before as connection time out or privilege issue that might have not interrupted the job but led to an empty input.

After you read the data, do you do any processing?

If there are no error and you can manage the code in PySpark, clone the job , open it in manual edit mode and then try to print the schema and show the content.

hope this helps,

EXPERT
answered 3 months ago
  • Yes, It is populated I found the problem, I didn't define schema for input and output data sources. I set it up and the job succeeded

    Thanks for your help!

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions