Glue Table Data Quality Jobs fail with java.lang.RuntimeException : Failed to fetch data.

The issue you're experiencing with AWS Glue Data Quality jobs failing on tables created via Spark SQL CTAS (Create Table As Select) commands in Glue Jobs is not a common scenario documented in the standard troubleshooting guides. However, based on the information provided, there are a few potential causes and solutions to consider:

1. Table Metadata: When tables are created using CTAS in a Glue Job, there might be discrepancies in how the metadata is stored or interpreted compared to tables created by crawlers or through the console. This could lead to issues when the Data Quality job tries to fetch data.

2. Permissions: Ensure that the IAM role used by the Data Quality job has the necessary permissions to access the data in the S3 location where the CTAS-created table data is stored.

3. Table Properties: Check if there are any specific table properties set during the CTAS operation that might be incompatible with Data Quality jobs.

4. Data Format: Verify that the data format used in the CTAS operation is fully compatible with Data Quality jobs.

5. Special Characters: If your table or column names contain special characters (like periods), this could cause issues with Data Quality jobs.

To troubleshoot and potentially resolve the issue:

1. Review the full Glue job logs and CloudWatch logs for any additional error messages or stack traces that might provide more context.

2. Try running the Data Quality job with a smaller subset of data to see if it completes successfully, which could help isolate whether it's a data volume issue.

3. Verify all connections and permissions, especially for the S3 bucket where the data is stored.

4. Consider creating a new table by copying data from the CTAS-created table using a method known to work with Data Quality jobs (e.g., using a crawler or console creation) and test if the Data Quality job succeeds on this new table.

5. If possible, modify your CTAS job to include additional metadata or table properties that might be expected by Data Quality jobs.

If the issue persists after trying these steps, it may be worth contacting AWS support to investigate if there's a potential bug or incompatibility between CTAS-created tables and Data Quality jobs. They might be able to provide more specific guidance or confirm if this is a known issue that needs to be addressed in the AWS Glue service.
Sources
Troubleshooting AWS Glue Data Quality errors - AWS Glue
Troubleshooting - AWS Glue
Debugging OOM exceptions and job abnormalities - AWS Glue