1 Answer
- Newest
- Most votes
- Most comments
0
To resolve the "Table or view already exists. SaveMode: ErrorIfExists" error when writing data to a PostgreSQL table using AWS Glue Spark, you need to specify the write mode as "append" instead of the default "ErrorIfExists" mode.
You can modify your code to:
trf_df.write.jdbc(url=jdbc_url, table='table_name', mode="append", properties=connection_properties)
The available write modes in Spark are:
- "append": Adds the data to the existing table without modifying the existing data
- "overwrite": Deletes the existing table and creates a new one (which you mentioned you don't want)
- "errorIfExists": Throws an error if the table exists (this is the default mode causing your error)
- "ignore": Silently ignores the operation if the table exists
Since you want to keep adding new data to your existing PostgreSQL table without deleting it, the "append" mode is the appropriate choice for your use case.
Sources
how to update database with glue etl job? | AWS re:Post
Relevant content
- asked 8 years ago
- asked 10 months ago
- asked 6 months ago
- asked 3 years ago
