getJDBCSink does not like Postgres timestamp fields


Hi All,
My DynamicFrame:

{"id": "fakeid1", "createdDate": "2020-11-28 20:20:44", "lastModifiedDate": "2029-10-26 17:23:40", "firstName": "yedUsFwdkelQbxeTeQOvaScfqIOOmaa", "middleName": "RYtGKbgicZaHCBRQDSx", "lastName": "JxkyvRnL", "secondLastName": "secondlastname", "prefix": "VLhpfQGTMDYpsBZxvfBoeygjb", "suffix": "UMaAIKKIkknjWEXJUfPxxQHeWKEJ", "role": "dpHYZGhtgdntugzvvKAXLhM", "title": "LlN", "department": "gNfZBdyFGRajVfJNonEnOinZj", "dateOfBirth": "2020-10-30", "sex": "UfzQhdgLLfDTDGspDb", "gender": "QvBQYuxiXXVytGCxzVllpgTJKhRQq", "pronoun": "qszYL", "kyc": true, "userStatus": "fakeuserstatus", "employeeNumber": "eOMtThyhVNLWUZNRcBaQKxI", "citizenId": "YdvDhtAsLghPXAgtbprXPZkhnfLTBSX"}   

My write command (which works for other tables):

glueContext.getJDBCSink(catalogConnection = catalogueConnection, options = JsonOptions(employeeAccountConnOptions), redshiftTmpDir = tempStorage).writeDynamicFrame(employeeAccountsDynF)  

Now, even though I can manually insert that string into a postgres db, as a timestamp field, glue will not. Glue says

org.postgresql.util.PSQLException : ERROR: column "createdDate" is of type timestamp without time zone but expression is of type character varying  

This typically means that the string I'm passing cannot be interpreted as a timestamp. That's why I tested it manually.

Any ideas? Please and thanks.

asked 2 years ago41 views
1 Answer

Found it. The spark column type was string. Cast it and it works.

answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions