How to escape a comma in a csv file in AWS Glue?
Customer is using Glue to parse a CSV file.
The issue happens when trying to load a CSV file and one of its text column EQUIPMENT_DESCRIPTION has “,” (Comma) value in it. This column value is identified with text qualifier (double quotes). But the way we currently have our crawler/glue configured, we are not able to parse the column with the text qualifier to ignore comma values in it.
This is causing all the other columns following EQUIPMENT_DESCRIPTION, to be filled with incorrect data including the DATE_COLLECTED column.
How do you configure Glue to ignore this comma?
You should confirm that the quote character is configured appropriately in the SerDe properties of the table?
Have a look at Working with CSV Files in Best Practices When Using Athena With AWS Glue.
Relevant questions
AWS glue combining multiple input into a single output csv
asked 3 months agoAWS Glue crawler detecting a .(dot) before header of a csv file
asked 3 years agoescape caracter in AWS glue
Accepted Answerasked 4 months agoAWS Glue read a csv file encoded in Windows 1252 with extended characters
Accepted AnswerHow could we have Glue to get data from csv as String?
Accepted Answerasked 7 days agoGlue processing a csv
Accepted Answerasked 2 years agoHow to escape a comma in a csv file in AWS Glue?
Accepted AnswerAmazon Athena query outputting all columns in the csv as VARCHARs
Accepted Answerasked 2 years agoError reading Kinesis table in Zeppelin
asked 3 months agoHow do I get the output of an AWS Glue DataBrew job to be a single CSV file?
Accepted Answerasked a year ago