Exception in User Class: org.postgresql.util.PSQLException : ERROR: column "id" does not exist

0

I'm doing ETL from Postgres to Redshift with AWS Glue. I have imported a table with a Crawler and created a Job to just transfer the data and create a new table in Redshift. I get: Exception in User Class: org.postgresql.util.PSQLException : ERROR: column "id" does not exist In the original table, the column is "Id", case sensitive. Is there a way to make Glue case sensitive? (I don't have permission to change the Postgres schema). Also, I tried to create a job with Spark 3.1, Scala 2 (Glue version 3.0) and set caseSensitive = false, still not working and getting the same error

질문됨 2년 전1802회 조회
1개 답변
0

Hi ,

I think that the issue is when you are reading the data from Postgres, because the query built by Glue uses the lower case column name. PostgreSQL is a case-sensitive database by default.

if you are using from_catalog to read the data, you could use the following code to avoid glue automatic query building, and force the query you need:

dyf_query_orddt = glueContext.create_dynamic_frame.from_catalog(
    database="mysqltest",
    table_name="classicmodels_orders",
    additional_options={"query":"SELECT * from classicmodels.orders WHERE orderDate > '2005-01-01';"},
    transformation_ctx = "dyf_query_orddt",

)

hope this helps

AWS
전문가
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠