- Newest
- Most votes
- Most comments
https://stackoverflow.com/questions/54468084/aws-glue-not-copying-idint-column-to-redshift-its-blank
When AWS Glue crawls a column, it might not decide on a definitive type for that column. Instead, it maintains a 'type choice' where the exact type of a crawled column can remain as a number of possibilities throughout the ETL Job. The crawler only crawls a subset of a column's data to determine the probable type. This is why when you run the job or look at a preview, the type of the column might get resolved to a type that is incompatible with the dataset, which results in empty data for the column in question.
The solution is to add into your script an explicit resolution of the choices, by casting the column that's failing to the desired target type. Here is how you can do it:
df.resolveChoice(specs = [('id', 'cast:int')])
Relevant content
- asked 2 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 5 months ago
I used Visual GUI create job, the script code no variable df