Boto3 - S3 - Downloads Occasionally failing

0

Hi Team,

I have written a python program that copies videos from S3 bucket to an EC2 machine, videos that are more than 100 MB in size. While the program runs fine most of the time, but sometimes the download of the file fails with the error, wherein the name of the file is changed to video_name+<hash_characters>. After doing wee bit of research I found out the boto3 library of python, downloads the file temp with a hash name and then replaces it with the name of the file once the download is completed.

Below is the log when the download fails :

: ERROR/ForkPoolWorker-3] Task api.service.push_to_async_report_generation[25fdce51-a067-4893-bf4c-22013368e59f] raised unexpected: FileNotFoundError(2, 'No such file or directory') Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 451, in trace_task R = retval = fun(*args, **kwargs) File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 734, in protected_call return self.run(*args, **kwargs) File "/home/ubuntu/src/api/service.py", line 100, in push_to_async_report_generation image_path = bucket.download_file(Kwargs["file_name"]) File "/home/ubuntu/src/utils/s3_services.py", line 81, in download_file self.bucket.download_file(file_name, os.path.join('./static',file_name)) File "/usr/local/lib/python3.8/dist-packages/boto3/s3/inject.py", line 244, in bucket_download_file return self.meta.client.download_file( File "/usr/local/lib/python3.8/dist-packages/boto3/s3/inject.py", line 170, in download_file return transfer.download_file( File "/usr/local/lib/python3.8/dist-packages/boto3/s3/transfer.py", line 307, in download_file future.result() File "/usr/local/lib/python3.8/dist-packages/s3transfer/futures.py", line 106, in result return self._coordinator.result() File "/usr/local/lib/python3.8/dist-packages/s3transfer/futures.py", line 265, in result raise self._exception File "/usr/local/lib/python3.8/dist-packages/s3transfer/tasks.py", line 126, in call return self._execute_main(kwargs) File "/usr/local/lib/python3.8/dist-packages/s3transfer/tasks.py", line 150, in _execute_main return_value = self._main(**kwargs) File "/usr/local/lib/python3.8/dist-packages/s3transfer/download.py", line 571, in _main fileobj.seek(offset) File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 367, in seek self._open_if_needed() File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 350, in _open_if_needed self._fileobj = self._open_function(self._filename, self._mode) File "/usr/local/lib/python3.8/dist-packages/s3transfer/utils.py", line 261, in open return open(filename, mode) FileNotFoundError: [Errno 2] No such file or directory: './static/example.mp4.F5e1dAFE

It seems like the S3transfer package is trying to open the file before it gets downloaded. Any help related to the topic is appreciated!

Thanks!

vineeth
asked 2 years ago484 views
2 Answers
0

This issue doesn't appear to be related to Storage Gateway. You should tag your question with S3 to get the correct eyes on your issue.

AWS
answered 2 years ago
0

The tag may be "Simple Storage Service" that you need

AWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions