Can Glue Spark timezone be changed?


We have upstream systems that all use Central US time zone, but our pyspark/sparkSQL jobs in Glue is UTC and current_timestamp() is giving UTC time. Can we direct glue to use a different timezone? We tried adding a configuration to SparkConf: ("spark.sql.session.timeZone", "America/Chicago")

We also tried adding --java-options -Duser.timezone="America/Chicago" from the dashboard Run with Parameters feature.

Neither had the effect of updating spark's timezone. Any help here?

asked a year ago1034 views
1 Answer

A timestamp doesn't have a timezone, by definition is based on UTC. The timezone you are configuring comes into play when you parse a date or you format that timestamp into a string.

If you do a "show()" on a timestamp column, you should see it in the timezone configured, if not maybe it's not correctly configured, notice that properly spark.sql.session.timeZone has to be set for SparkSession, not context.

profile pictureAWS
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions