Hi Team,
I have written a python program that copies videos from S3 bucket to an EC2 machine, videos that are more than 100 MB in size. While the program runs fine most of the time, but sometimes the download of the file fails with the error, wherein the name of the file is changed to video_name+<hash_characters>.
After doing wee bit of research I found out the boto3 library of python, downloads the file temp with a hash name and then replaces it with the name of the file once the download is completed.
Below is the log when the download fails :
: ERROR/ForkPoolWorker-3] Task api.service.push_to_async_report_generation[25fdce51-a067-4893-bf4c-22013368e59f] raised unexpected: FileNotFoundError(2, 'No such file or directory')
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in protected_call
return self.run(*args, **kwargs)
File "/home/ubuntu/src/api/service.py", line 100, in push_to_async_report_generation
image_path = bucket.download_file(Kwargs["file_name"])
File "/home/ubuntu/src/utils/s3_services.py", line 81, in download_file
self.bucket.download_file(file_name, os.path.join('./static',file_name))
File "/usr/local/lib/python3.8/dist-packages/boto3/s3/inject.py", line 244, in bucket_download_file
return self.meta.client.download_file(
File "/usr/local/lib/python3.8/dist-packages/boto3/s3/inject.py", line 170, in download_file
return transfer.download_file(
File "/usr/local/lib/python3.8/dist-packages/boto3/s3/transfer.py", line 307, in download_file
future.result()
File "/usr/local/lib/python3.8/dist-packages/s3transfer/futures.py", line 106, in result
return self._coordinator.result()
File "/usr/local/lib/python3.8/dist-packages/s3transfer/futures.py", line 265, in result
raise self._exception
File "/usr/local/lib/python3.8/dist-packages/s3transfer/tasks.py", line 126, in call
return self._execute_main(kwargs)
File "/usr/local/lib/python3.8/dist-packages/s3transfer/tasks.py", line 150, in _execute_main
return_value = self._main(**kwargs)
File "/usr/local/lib/python3.8/dist-packages/s3transfer/download.py", line 571, in _main
fileobj.seek(offset)
File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 367, in seek
self._open_if_needed()
File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 350, in _open_if_needed
self._fileobj = self._open_function(self._filename, self._mode)
File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 261, in open
return open(filename, mode)
FileNotFoundError: [Errno 2] No such file or directory: './static/example.mp4.F5e1dAFE
It seems like the S3transfer package is trying to open the file before it gets downloaded.
Any help related to the topic is appreciated!
Thanks!