AWS Glue Jobs 3.0 Unable to parse CSV file

Question

Hello, I am running a job to apply an ETL on a semi-colon-separated CSV on S3, however, when I read the file using the DynamicFrame feature of AWS and try to use any method like `printSchema` or `toDF`, I get the following error:

```
py4j.protocol.Py4JJavaError: An error occurred while calling o77.schema.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (52bff5da55da executor driver): com.amazonaws.services.glue.util.FatalException: Unable to parse file: s3://my-bucket/my-file.csv
```
I have already verified the codification, it is UTF-8 so there should be no problem. When I read the CSV using `spark.read.csv`, it works fine, and the Crawlers can also recognize the schema. The data has some special characters that shouldn't be there, and that's part of the ETL I am looking to perform. Neither using the `from_catalog` nor `from_options` function from AWS Glue works, the problem is the same whether I run the job locally on docker or Glue Studio.  My data have a folder date partition so I would prefer to avoid using directly Spark to read the data and take advantage of the Glue Data Catalog as well.

Thanks in advance.

Accepted Answer

Hello,

Error "Fatal exception com.amazonaws.services.glue.readers unable to parse file *.csv" is observed when CSV is either not "UTF-8" encoded or if it has non UTF-8 encoded characters.
~~~~
As you have already verified that your CSV file is UTF-8 encoded and you mentioned that data has some special characters, please try running ETL job after removing these characters from file.

Using a semi colon separated CSV file I have tested in my personal AWS account, I was able to run Glue ETL job successfully using the from_catalog function.
Steps I followed for this test:
1. 	Created a sample semi colon seprated CSV file.
2. 	Created the table from this file using Glue Crawler.
3. 	Created a Glue ETL job to read this table using from_catalog function

AWS Glue Jobs 3.0 Unable to parse CSV file

Relevant content