"Error Category: UNCLASSIFIED_ERROR; An error occurred while calling z:com.amazonaws.services.glue.DynamicFrame.apply. YEAR"

The error you're encountering, "UNCLASSIFIED_ERROR; An error occurred while calling z:com.amazonaws.services.glue.DynamicFrame.apply. YEAR", suggests an issue with applying a transformation or operation on a DynamicFrame in AWS Glue.

Given that the error occurs only when you include a specific join operation, it's likely that there's a problem with the data or schema in that particular join. The mention of "YEAR" in the error message hints at a possible issue related to a date or timestamp field.

Here are some steps you can take to troubleshoot and potentially resolve the issue:

1. Check the schema of the tables involved in the problematic join, particularly any date or timestamp fields. Ensure that the data types are consistent and compatible.

2. Examine the data quality in the tables, especially in the columns used for joining. Look for any null values, unexpected formats, or inconsistencies that might cause issues during the join operation.

3. Use the errorsAsDynamicFrame() method on your DynamicFrame to get more detailed information about the errors. This will return a new DynamicFrame containing nested records for errors that occurred during the creation or transformation of your source DynamicFrame.

4. If possible, try to isolate the problematic join by creating a smaller subset of your data and testing it separately. This can help identify if the issue is with the data itself or with the complexity of multiple joins.

5. Consider using the resolveChoice() method on your DynamicFrames before joining if you have columns with inconsistent types across rows.

6. Review your join conditions and ensure they are correctly specified, especially if you're using any date or time-based joins.

7. If the error persists, you may need to increase the resources allocated to your Glue job, as complex joins with large datasets can be resource-intensive.

If these steps don't resolve the issue, you might need to collect more detailed logs and potentially reach out to AWS Support for further assistance, especially if you're dealing with a large-scale or complex ETL process.
Sources
DynamicFrame class - AWS Glue
ErrorsAsDynamicFrame class - AWS Glue

Hi, Can you post the join code to assist, please use some dummy names and remove any unnecessary logic. This error is happening at Apply transformation and only happening with YEAR field/column suggests there is some format or data error. I'd recommend access this YEAR field alone in a separate dynamicFrame with the format or transformation you're using and try to write to a temp location.