How Can I extract last 8 characters of a string which represent the date in this format 'YYYYMMDD' and convert into date?

0

Hi there,

I have a field that contain either a '"/" or "_" and after these characters are characters which represent the date in the format 'YYYYMMDD' How can I extract them an cast into date format?

Many thanks,

JD

  • Hi which service do you plan to use to work with the data?

JD
asked 2 years ago1004 views
1 Answer
1

From your question, it is not really clear where the data is stored and what tools you are using to access the data. Depending on where the data is available (Redshift vs RDS vs S3 vs Kinesis etc) and depending on the tool used to query the data, the answer would change.

I will list a couple of ways you can accomplish this:

  1. In Athena when the data is cataloged in Glue tables (Eg. the data is in S3): you could execute this query - select cast(date_parse('20220915','%Y%m%d') as date);
  2. In Glue when you are accessing the data as a Glue dynamicFrame or Pyspark dataframe:
from pyspark.sql.functions import *
#df is the dataframe that you add logic here to read/transform or convert from Glue Dynamicframe
df.select(col("input"),to_date(col("input"),"yyyyMMdd").alias("date_column")).show()
  1. In Redshit: select to_date('20220915', 'YYYYMMDD', FALSE);

Reference: https://docs.aws.amazon.com/redshift/latest/dg/r_TO_DATE_function.html

profile pictureAWS
answered 2 years ago
AWS
EXPERT
reviewed 2 years ago
  • Hi there,

    Thanks so much for your help. So I'm using AWS to query the database and I have tried option 1 with no success. Error message reads as:

    INVALID_FUNCTION_ARGUMENT: Invalid format: ""

    The field I am trying to perform this operation on is a result of a substring function which extracts the last 8 characters of another field that contains the date as part of a longer unusable string for date purposes. The result of the substring is in 'YYYMMDD' format.

    Thanks again

  • @JD is the format "YYYYMMDD" or "YYYMMDD", I think you have a typo in the comment. aside from that could you provide your athena query? and some info on the table format (DDL)?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions