Why does AWS Glue SAP OData Connector fail to extract nullable fields with null values?

This issue appears to be related to how AWS Glue handles fields with null values during schema inference. When using DynamicFrames (which the SAP OData Connector uses), AWS Glue infers the schema from the actual data rather than using a predefined catalog. If all values in a field are null, the field may be omitted from the schema even if explicitly specified in SELECTED_FIELDS.

This is a known behavior with AWS Glue's DynamicFrames - they infer schema from the actual data files, and fields containing only null values may be inconsistently detected or omitted entirely.

To address this issue, you could try several approaches:

1. Convert to DataFrame: After extracting data with the connector, convert your DynamicFrame to a DataFrame with an explicitly defined schema that includes all expected fields.

2. Define explicit schema: Create a complete schema definition that includes all fields (including nullable ones) and apply it to your data.

3. Add placeholder values: If possible, ensure at least one non-null value exists in these fields in your source data.

4. Use alternative connection methods: Some users have reported more reliable schema detection using different connection approaches.

5. Implement schema validation: Add validation checks in your Glue job to ensure all expected fields are present and add missing ones if needed.

The DropNullFields transform in AWS Glue is designed to remove fields with null values, but in your case, you need the opposite - to retain fields even when they contain all null values. This is a limitation in how schema inference works with DynamicFrames rather than an issue specific to the SAP OData Connector.
Sources
AWS Glue - Missing fields after extraction | AWS re:Post
DropNullFields class - AWS Glue