Writing into redshift with Glue 4.0 fails due to string lengths

Question

I'm writing into redshift and realized Glue 4.0 is probably optimizing the column sizes. 
Summary of error:

```
py4j.protocol.Py4JJavaError: An error occurred while calling o236.pyWriteDynamicFrame.
: java.sql.SQLException: 
Error (code 1204) while loading data into Redshift: "String length exceeds DDL length"
Table name: "PUBLIC"."table_name"
Column name: column_a
Column type: varchar(256)
```

In previous glue versions, the string columns were always varchar(65535) but now, my tables are created with varchar(256), and writing into some columns fail due to this error. Now, will this occur with other data types? . How can I solve this within Glue 4.0?

mugiwara · Answer

The closest answer I've found is about the new redshift driver for spark under the 'Configuring the maximum size of string columns
': https://docs.databricks.com/external-data/amazon-redshift.html#language-python

But this is with respect to spark. How can I translate this to Glue?

Jhon Torres · Answer

I found this document, but I don't know how to apply it to my visual ETL in Glue https://docs.aws.amazon.com/redshift/latest/mgmt/spark-redshift-connector-other-config.html

Jhon Torres · Answer

I found this: [https://stackoverflow.com/questions/77382178/glue-job-failed-when-trying-to-perform-a-merge-into-redshift-table-due-to-string] the solution is to edit the script and change the length of the varchar column when creating the temporary table.

Writing into redshift with Glue 4.0 fails due to string lengths

Relevant content