I believe PostgreSQL column names are case-sensitive(when double quoted as given in the code). AWS Glue does not support using case-sensitive columns as job bookmark keys. I think that is why glue is still using primary key as the job bookmark key. Check out the documentation for the same when using with JDBC sources https://docs.aws.amazon.com/glue/latest/dg/monitor-continuations.html
- asked 4 months ago
- Accepted Answerasked 10 months ago
- Why does my AWS Glue job fail with lost nodes when I migrate a large data set from Amazon RDS to Amazon S3?AWS OFFICIALUpdated 6 months ago
- AWS OFFICIALUpdated 4 months ago
- What are the best practices to use when migrating PostgreSQL databases to a target RDS for PostgreSQL database using AWS DMS?AWS OFFICIALUpdated 7 months ago
- AWS OFFICIALUpdated 10 months ago
- A Brief Primer to Onboarding Data To a Healthcare and Life Sciences Data Mesh Leveraging AWS ServicesEXPERTpublished 7 months ago
- EXPERTpublished 7 months ago
Thank you so much for your reply. I am confuse whether it is possible to change it to non-case-sensitive (with single quoted? ), and make it work for PostgreSQL or it is impossible or it is not possible to work with PostgreSQL?
yes of course it is possible to work with PostgreSQL. As per your code you are using 'additional_options' param in 'create_dynamic_frame.from_options' function which is not correct way. If you look into the documentation link, you would notice this - You can specify jobBookmarkKeys and jobBookmarkKeysSortOrder in the following ways: create_dynamic_frame.from_catalog — Use additional_options. create_dynamic_frame.from_options — Use connection_options. you can try giving the same in "connection_options" as mentioned and check if it works for you.
Alternatively, you can also try another way - create a glue crawler to crawl your PostgreSQL datastore and run it to create glue catalog metadata table. you can pass in classifiers in crawler config or even map columns to different name/type using applyMapping feature. Then create the dynamic frame using 'glueContext.create_dynamic_frame.from_catalog' function and pass in bookmark keys in 'additional_options' param.
Thank you so much for the help. Move the arguments to 'connection_options' works