1 Answer
- Newest
- Most votes
- Most comments
1
- Glue does automatically creates the table schema is it doesn't exist.
However, any "string" type will be translated into a
varchar(65535)
, which is not efficient for a columnar database. - If the table already exists, Glue will just add the records into the table.
- If Glue can't map a field into the table (example: you have a field "createdate" as string type in your python code and the table has a "createdate" as timestamp type in Redshift), Glue will automatically add a field "createdate_string" to the table and populate that field.
When Glue calls glueContext.write_dynamic_frame.from_jdbc_conf()
, it stores the output dataset into the S3 bucket given by: redshift_tmp_dir = "s3://jm-bank/tmp/"
. Then Glue performs a COPY command to Redshift, which you can visualize in "Queries" option from your Redshift cluster.
answered 6 years ago
Relevant content
- Accepted Answerasked 2 years ago
- asked a year ago
- How can I use a Lambda function to automatically start an AWS Glue job when a crawler run completes?AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago
If the table exists, has all the same columns, but the
varchar
columns are smaller thanvarchar(65535)
does Glue automatically rebuild the table?My tables go back to having
varchar(65535)
and seem to be missing distribution style/key and sort keys that I added just a couple of days ago.