aws glue create_dynamic_frame from data in PostgreSQL with custom bookmark key

Question

Hi AWS expert, I have a code read data from AWS aurora PostgreSQL, I want to bookmark the table with custom column named  'ceres_mono_index'. But it seems like the bookmark is still uses the primary key as the bookmark key instead of column 'ceres_mono_index'. Here is the code

```python

cb_ceres = glueContext.create_dynamic_frame.from_options(
    connection_type="postgresql",
    connection_options={
        "url": f"jdbc:postgresql://{ENDPOINT}:5432/{DBNAME}",
        "dbtable": "xxxxx_raw_ceres",
        "user": username,
        "password": password,
    },
    additional_options={"jobBookmarkKeys": ["ceres_mono_index"], "jobBookmarkKeysSortOrder": "asc"},
    transformation_ctx="cb_ceres_bookmark",
)

```

How could I fix the issue? Thank you

Answer

I believe PostgreSQL column names are case-sensitive(when double quoted as given in the code).  AWS Glue does not support using case-sensitive columns as job bookmark keys. I think that is why glue is still using primary key as the job bookmark key.
Check out the documentation for the same when using with JDBC sources
https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html

aws glue create_dynamic_frame from data in PostgreSQL with custom bookmark key

Relevant content