내용으로 건너뛰기

Table or view already exists. SaveMode: ErrorIfExists while writing the data to postgress table using glue spark

0

Hi Team,

I'm Getting the below error while writing the table in postgres using glue spark script can you please help me on this issue.

"Table or view "asset_aircraft_rpt" already exists. SaveMode: ErrorIfExists.,"

Using the below code: trf_df.write.jdbc(url=jdbc_url, table='table_name', properties=connection_properties)

if we use the overwrite mode, the table is deleting and creating again. we want to load the data to postgress table form the glue scripts without deleting th table. please share the suggestions for the issue.

질문됨 일 년 전157회 조회
1개 답변
0

To resolve the "Table or view already exists. SaveMode: ErrorIfExists" error when writing data to a PostgreSQL table using AWS Glue Spark, you need to specify the write mode as "append" instead of the default "ErrorIfExists" mode.

You can modify your code to:

trf_df.write.jdbc(url=jdbc_url, table='table_name', mode="append", properties=connection_properties)

The available write modes in Spark are:

  • "append": Adds the data to the existing table without modifying the existing data
  • "overwrite": Deletes the existing table and creates a new one (which you mentioned you don't want)
  • "errorIfExists": Throws an error if the table exists (this is the default mode causing your error)
  • "ignore": Silently ignores the operation if the table exists

Since you want to keep adding new data to your existing PostgreSQL table without deleting it, the "append" mode is the appropriate choice for your use case.
Sources
how to update database with glue etl job? | AWS re:Post

답변함 일 년 전
AWS
지원 엔지니어
검토됨 일 년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.