Skip to content

Table or view already exists. SaveMode: ErrorIfExists while writing the data to postgress table using glue spark

0

Hi Team,

I'm Getting the below error while writing the table in postgres using glue spark script can you please help me on this issue.

"Table or view "asset_aircraft_rpt" already exists. SaveMode: ErrorIfExists.,"

Using the below code: trf_df.write.jdbc(url=jdbc_url, table='table_name', properties=connection_properties)

if we use the overwrite mode, the table is deleting and creating again. we want to load the data to postgress table form the glue scripts without deleting th table. please share the suggestions for the issue.

asked 10 months ago142 views
1 Answer
0

To resolve the "Table or view already exists. SaveMode: ErrorIfExists" error when writing data to a PostgreSQL table using AWS Glue Spark, you need to specify the write mode as "append" instead of the default "ErrorIfExists" mode.

You can modify your code to:

trf_df.write.jdbc(url=jdbc_url, table='table_name', mode="append", properties=connection_properties)

The available write modes in Spark are:

  • "append": Adds the data to the existing table without modifying the existing data
  • "overwrite": Deletes the existing table and creates a new one (which you mentioned you don't want)
  • "errorIfExists": Throws an error if the table exists (this is the default mode causing your error)
  • "ignore": Silently ignores the operation if the table exists

Since you want to keep adding new data to your existing PostgreSQL table without deleting it, the "append" mode is the appropriate choice for your use case.
Sources
how to update database with glue etl job? | AWS re:Post

answered 10 months ago
AWS
SUPPORT ENGINEER
reviewed 10 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.