An error occurred while calling o492.pyWriteDynamicFrame. Cannot cast STRING into a NullType (value: BsonString{value='72667'})

py4j.protocol.Py4JJavaError: An error occurred while calling o492.pyWriteDynamicFrame. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 323 in stage 2.0 failed 4 times, most recent failure: Lost task 323.3 in stage 2.0 (TID 356) (172.31.17.150 executor 1): com.mongodb.spark.exceptions.MongoTypeConversionException: Cannot cast STRING into a NullType (value: BsonString{value='72667'}).

During the execution of my Glue job, I encountered an error specifically when attempting to read data from a particular table in MongoDB. It is important to note that I have been able to successfully read data from 30 other tables in MongoDB using Glue without any issues.

To address this problem, I attempted several approaches in order to retrieve the data from the problematic table. First, I tried changing all the data types to strings and increased the sample size. Despite these efforts, the Glue job failed after reading approximately 70 to 80% of the data.

In my attempt to overcome this issue, I utilized the Glue Data Catalog and created a crawler to fetch the data from MongoDB. However, even with this approach, I encountered the same error.

To further troubleshoot the problem, I also tried reading the data from the MongoDB table using a simple PySpark code, ensuring that all the necessary connection details were provided. Unfortunately, this alternative approach also proved to be unsuccessful.

In my effort to address the problem, I searched for potential solutions and came across a relevant Stack Overflow page. https://stackoverflow.com/questions/69219209/aws-glue-error-an-error-occurred-while-calling-o75-pywritedynamicframe-cannot

I followed the suggestions provided on that page, hoping to resolve the issue. Unfortunately, despite implementing the recommended solution, I have been unable to overcome the problem I am facing with this particular table.