AWS Glue - Missing fields after extraction

The issue you're experiencing with missing fields in your AWS Glue extraction job from MongoDB is a known problem, particularly when dealing with fields that contain null values. This inconsistency in schema detection can occur when using DynamicFrames, as they infer the schema from the actual data files rather than using a predefined catalog.

To address this issue, you have several options:

1. Convert to DataFrame: Instead of using DynamicFrames, consider converting your data to a DataFrame. DataFrames use the catalog and may provide more consistent results in schema detection.

2. Use explicit schema definition: If possible, define the schema explicitly based on a representative sample of your data, and then apply it to the full dataset.

3. Implement error handling: Add retry logic and error handling to your code to account for temporary connection issues or data inconsistencies.

4. Consider alternative connection methods: Try connecting to MongoDB using a URI string, username, and password instead of the AWS Glue connector. Some users have reported this approach to be more reliable.

5. Update connectors: Ensure you're using the latest version of the JDBC connector for MongoDB, as newer versions may include bug fixes and improvements for schema detection.

6. Pre-process data: If feasible, consider pre-processing or flattening your MongoDB data before ingesting it into Glue. This can help ensure a more consistent schema.

7. Implement schema validation: Add schema validation checks in your Glue job to ensure all expected fields are present before processing the data.

If null values are indeed causing issues, you might also want to consider using the DropNullFields transform in AWS Glue. This transform can remove fields from the dataset if all values in the field are null. However, be aware that by default, AWS Glue only recognizes null objects and not empty strings, "null" strings, or other custom null values.

If these solutions don't resolve the issue, you may want to consider alternative ETL tools or approaches that are better suited for handling MongoDB data with complex or variable schemas. Additionally, reporting the issue to AWS support may help in getting a more permanent solution if it's indeed a bug in the Glue connector for MongoDB.
Sources
AWS Glue Notebook Cannot Identify Schema Consistently for MongoDB Atlas Source | AWS re:Post
Using DropNullFields to remove fields with null values - AWS Glue