Exception in User Class: org.postgresql.util.PSQLException : ERROR: column "id" does not exist

0

I'm doing ETL from Postgres to Redshift with AWS Glue. I have imported a table with a Crawler and created a Job to just transfer the data and create a new table in Redshift. I get: Exception in User Class: org.postgresql.util.PSQLException : ERROR: column "id" does not exist In the original table, the column is "Id", case sensitive. Is there a way to make Glue case sensitive? (I don't have permission to change the Postgres schema). Also, I tried to create a job with Spark 3.1, Scala 2 (Glue version 3.0) and set caseSensitive = false, still not working and getting the same error

已提問 2 年前檢視次數 1802 次
1 個回答
0

Hi ,

I think that the issue is when you are reading the data from Postgres, because the query built by Glue uses the lower case column name. PostgreSQL is a case-sensitive database by default.

if you are using from_catalog to read the data, you could use the following code to avoid glue automatic query building, and force the query you need:

dyf_query_orddt = glueContext.create_dynamic_frame.from_catalog(
    database="mysqltest",
    table_name="classicmodels_orders",
    additional_options={"query":"SELECT * from classicmodels.orders WHERE orderDate > '2005-01-01';"},
    transformation_ctx = "dyf_query_orddt",

)

hope this helps

AWS
專家
已回答 2 年前

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南