file not found error

Question

Hello,

Im trying with EMR Serverless, switching from EMR getting below error

```
Traceback (most recent call last):
  File "/tmp/spark-07b8331c-566b-43e9-bf1a-f0a77cb86420/spark_main.py", line 7, in 
    from app.parsers.core_signal_parser import CoreSignalDataParser
  File "", line 983, in _find_and_load
  File "", line 967, in _find_and_load_unlocked
  File "", line 668, in _load_unlocked
  File "", line 638, in _load_backward_compatible
  File "/tmp/spark-07b8331c-566b-43e9-bf1a-f0a77cb86420/sources_5dbc92a.zip/app/parsers/core_signal_parser.py", line 29, in 
  File "", line 983, in _find_and_load
  File "", line 967, in _find_and_load_unlocked
  File "", line 668, in _load_unlocked
  File "", line 638, in _load_backward_compatible
  File "/tmp/spark-07b8331c-566b-43e9-bf1a-f0a77cb86420/sources_5dbc92a.zip/app/formatter/candidate_formatter.py", line 11, in 
  File "", line 983, in _find_and_load
  File "", line 967, in _find_and_load_unlocked
  File "", line 668, in _load_unlocked
  File "", line 638, in _load_backward_compatible
  File "/tmp/spark-07b8331c-566b-43e9-bf1a-f0a77cb86420/sources_5dbc92a.zip/app/parsers/general_parser.py", line 23, in 
  File "", line 983, in _find_and_load
  File "", line 967, in _find_and_load_unlocked
  File "", line 668, in _load_unlocked
  File "", line 638, in _load_backward_compatible
  File "/tmp/spark-07b8331c-566b-43e9-bf1a-f0a77cb86420/sources_5dbc92a.zip/app/core/config.py", line 120, in 
  File "/tmp/spark-07b8331c-566b-43e9-bf1a-f0a77cb86420/sources_5dbc92a.zip/app/core/config.py", line 86, in __init__
FileNotFoundError: [Errno 2] No such file or directory: 'config.json'
```
I had tried to pass files with `--files s3://{S3_BUCKET}/emr/pyspark/code/users/devops/processing_source_code/config.json` No luck, can anyone help with this

Accepted Answer

Hello there,

I understand that you want to read the file which has been passed to EMR serverless using --files or --conf spark.files. In EMR serverless, the files are placed in a temporary location which is not the python execution location hence the error.

A workaround for the issue in EMR serverless is to use [pyspark.SparkFiles.get](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkFiles.get.html)

An example code snippet on how to use it is as below.

```
import pyspark
from pyspark.sql import SparkSession
from pyspark import SparkFiles
spark = SparkSession.builder \
                    .appName('').enableHiveSupport() \
                    .getOrCreate()

path = SparkFiles.get("config.json")
f = open(path, "r")
print(f.read())
spark.stop()
```

i hope the above resolves your issue and please revert if the workaround is not working for you.

Have a great day ahead.

file not found error

相關內容