S3 Trigger to HTTP Post "Device Busy"

0

ERROR!!: [Errno 16] Device or resource busy

import json
import os
import boto3
import base64
import http.client
import io
import urllib.parse

s3_client = boto3.client('s3')

def lambda_handler(event, context):
    con = None
    try:
        #Obtain S3 Object. Body = botocore.response.StreamingBody object
        print('TRANSPORT STREAM EVENT')
        s3_Bucket_Name = event["Records"][0]["s3"]["bucket"]["name"]
        s3_File_Name = event["Records"][0]["s3"]["object"]["key"]
        print(s3_Bucket_Name)
        print(s3_File_Name)
        s3_response = s3_client.get_object(Bucket=s3_Bucket_Name, Key=s3_File_Name)
        print(s3_response)

        auth_bytes = os.environ.get("AUTH").encode('utf-8')
        headers = {
            'Content-type': 'application/x-www-form-urlencoded',
            'Authorization': 'Basic {}'.format(base64.b64encode(auth_bytes).decode('utf-8'))
        }
        # Set the Content-Length header
        headers['Content-Length'] = str(s3_response['ContentLength'])

        rootFolder = os.environ.get("STREAM_FOLDER")

        params = {
            "path": rootFolder,
            "fileName": s3_File_Name,
            "searchable": "false",
            "crossDomain": "true"
        }
        query_string = '&'.join([f"{key}={value}" for key, value in params.items()])
        testUrl = os.environ.get("TEST_URL")
        uploadContext = os.environ.get("UPLOAD_CONTEXT")
        print("POST TO:: " + testUrl + uploadContext + '?' + query_string)
        encoded_params = urllib.parse.urlencode(params)

        streaming_body = s3_response['Body']

        con = http.client.HTTPConnection(testUrl, 80, timeout=10)
        con.request('POST', uploadContext + '?' + encoded_params, body=streaming_body, headers=headers)
        response = con.getresponse()
        print("TEST Response:: " + response.read().decode())
    except OSError as e:
        print(f"ERROR!!: {e}")

    finally:
        if con is not None:
            con.close()
  • You have a lot in that try clause - what happens after the first get_object call - when you print the response print(s3_response) Would be nice to know what bits of the code is working and what is failing?

cobar79
gefragt vor einem Monat118 Aufrufe
1 Antwort
0

Hello.

Execution environments may be reused when the same Lambda is used continuously.
As a result, I think there may be resource contention or something like that.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html#runtimes-lifecycle

Are there any other detailed error logs output?
Although it is difficult to determine without looking at detailed logs, an error may occur if there is a conflict between HTTP connections.
Why not try explicitly closing the HTTP connection with "con.close()"?
I thought that closing the HTTP connection at the end of processing would reduce the possibility of resource contention even if the environment was reused.

profile picture
EXPERTE
beantwortet vor einem Monat
profile picture
EXPERTE
überprüft vor einem Monat
  • I appreciate the quick response. No there isn't much detail to the logging. Yes, I had explicit close before putting in the except. I was hoping to get more detail printing the exception, still trying to figure out how to get debug level logs.

    My best guess was the <botocore.response.StreamingBody> resource. So, I tried reading the stream into memory ~200MB ("ts" media archive file), close the StreamBody and post the Byte array, but got the same thing.

    That led me to believe it was the client not handling the stream, the stream pointer issue, need to chunk the reading of the stream so I tried to use the more current requests and urllib3 clients, but they are not available and I don't want to go through the layer process to find out that is not the issue. I would rather convert from Python to Java to see if I can get information on what the "Device" is referring to. That an I am a once in a few years Python coder so I am not overly familiar with the Python libraries.

    I ruled out a network issue since I have another java microservice posting JSON to the same gateway and I didn't get an http error.

    I just thought it would be a common easy thing to read s3 object and post using Python.

  • Basically it came down to the client post. I have abandoned the Python version and created a Spring Function "Function<S3Event, Void>" that successfully posted the video TS file without reading the stream into memory.

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen