glue job fail for file encoding

Question

Hi team,

I have a glue job that read from an S3 CSV file and inject it to DB,

I have the following error while running the job,

I think it's related to the file encoding,

the original file encoding is : ** ISO-8859-1**

if I change manually the file encoding to be UTF-8, the glue job passes.

do I need to have the CSV file encoded in utf-8 to be able to run successfully the job way?
is there any way to go around this?

Thank you!

```
com.amazonaws.services.glue.util.FatalException: Unable to parse file: myFile.csv

at com.amazonaws.services.glue.readers.JacksonReader.hasNextFailSafe(JacksonReader.scala:94)
	at com.amazonaws.services.glue.readers.JacksonReader.hasNext(JacksonReader.scala:38)
	at com.amazonaws.services.glue.readers.CSVReader.hasNext(CSVReader.scala:169)
	at com.amazonaws.services.glue.hadoop.TapeHadoopRecordReaderSplittable.nextKeyValue(TapeHadoopRecordReaderSplittable.scala:97)
	at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:247)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at
```

Answer

Hi - You need to convert the character encoding from ISO-8859-1 to UTF-8 before letting AWS Glue process it. 
 
https://docs.aws.amazon.com/glue/latest/dg/components-key-concepts.html
> Text-based data, such as CSVs, must be encoded in UTF-8 for AWS Glue to process it successfully.

There are few examples listed here -https://github.com/aws-samples/aws-glue-samples/blob/master/examples/converting_char_encoding.md which use spark to convert the datatype.

glue job fail for file encoding

相关内容