Skip to content

[Glue job]Connect with RDS using jdbc url including character_set_server encoding parameter is not working

0

I need to insert a dynamicFrame created by reading a table from Redshift to RDS and the frame includes a column has string values in Korean. My target DB's default encoding is not utf-8, so I needed to pass an encoding option when writing the frame. I tried below code but found Korean letters are still not readable in mySQL.

What is the reason the jdbc url with encoding parameters didn't work as I intended?

Code

write_frame = glueContext.write_dynamic_frame.from_jdbc_conf(
    frame = source_frame,
    connection_options = {
        "url": "jdbc:mysql://<mysql host address>:<mysql port>/<db>?useUnicode=true&character_set_server=utf8mb4&character_set_client=utf8mb4&character_set_connection=utf8mb4&character_set_database=utf8mb4&character_set_results=utf8mb4&character_set_system=utf8mb4",
        "database": args["mysql_database"],
        "dbtable": target_table,
        "connectionName": args["mysql_connection_name"],
        "aws_iam_role": args["mysql_iam_role"]
    }
)

job.commit()

Result: the string shows as ? instead of the correct Korean letters.

  • Is it possible that the characters are already mixed up ? at that point? Have you tried to do a source_frame.show() and check in the output look it is correct before saving

  • @Gonzalo Herreros

    The source dynamic frame was ok, I inserted the same frame into redshift before insert it into RDS and checked the string values are shown correctly in Redshift.

  • That will just use the standard MySQL connector, try using Glue 4.0 so you have a recent version of the driver. Have you ever tried to use those flags with MySQL JDBC before?

  • @Gonzalo Herreros Thanks for the suggestion. This is first time to use MySQL JDBC in Glue job. Could you advise which updates in Glue 4.0 makes recognize unicode & charset parameters in jdbc url that Glue 3.0 couldn't recognize? For your convienence, if there is any doc I can reference, just sharing it's link would be really helpful.

asked 3 years ago696 views
1 Answer
1
Accepted Answer

That doesn't really depend on Glue, Glue brings the MySQL JDBC driver 8.0.23 (actually it is the same as Glue 3) and anything you pass in the url has to be supported by the driver: https://dev.mysql.com/doc/connector-j/8.0/en/connector-j-reference-configuration-properties.html#table-connector-j-connp-props-session
Notice the name of the parameters is not exactly the same as the properties when you set then inside the database

AWS
EXPERT
answered 3 years ago
  • Thanks, your answer gave me a hint. I solved the problem by using the updated version of JDBC driver. Fyr, as I checked, Glue 4.0 and Glue 3.0 supports a same version of J driver so I downloaded the upper version from MySQL site.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.