S3 Trigger to HTTP Post "Device Busy"

0

ERROR!!: [Errno 16] Device or resource busy

import json
import os
import boto3
import base64
import http.client
import io
import urllib.parse

s3_client = boto3.client('s3')

def lambda_handler(event, context):
    con = None
    try:
        #Obtain S3 Object. Body = botocore.response.StreamingBody object
        print('TRANSPORT STREAM EVENT')
        s3_Bucket_Name = event["Records"][0]["s3"]["bucket"]["name"]
        s3_File_Name = event["Records"][0]["s3"]["object"]["key"]
        print(s3_Bucket_Name)
        print(s3_File_Name)
        s3_response = s3_client.get_object(Bucket=s3_Bucket_Name, Key=s3_File_Name)
        print(s3_response)

        auth_bytes = os.environ.get("AUTH").encode('utf-8')
        headers = {
            'Content-type': 'application/x-www-form-urlencoded',
            'Authorization': 'Basic {}'.format(base64.b64encode(auth_bytes).decode('utf-8'))
        }
        # Set the Content-Length header
        headers['Content-Length'] = str(s3_response['ContentLength'])

        rootFolder = os.environ.get("STREAM_FOLDER")

        params = {
            "path": rootFolder,
            "fileName": s3_File_Name,
            "searchable": "false",
            "crossDomain": "true"
        }
        query_string = '&'.join([f"{key}={value}" for key, value in params.items()])
        testUrl = os.environ.get("TEST_URL")
        uploadContext = os.environ.get("UPLOAD_CONTEXT")
        print("POST TO:: " + testUrl + uploadContext + '?' + query_string)
        encoded_params = urllib.parse.urlencode(params)

        streaming_body = s3_response['Body']

        con = http.client.HTTPConnection(testUrl, 80, timeout=10)
        con.request('POST', uploadContext + '?' + encoded_params, body=streaming_body, headers=headers)
        response = con.getresponse()
        print("TEST Response:: " + response.read().decode())
    except OSError as e:
        print(f"ERROR!!: {e}")

    finally:
        if con is not None:
            con.close()
  • You have a lot in that try clause - what happens after the first get_object call - when you print the response print(s3_response) Would be nice to know what bits of the code is working and what is failing?

cobar79
已提問 1 個月前檢視次數 119 次
1 個回答
0

Hello.

Execution environments may be reused when the same Lambda is used continuously.
As a result, I think there may be resource contention or something like that.
https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html#runtimes-lifecycle

Are there any other detailed error logs output?
Although it is difficult to determine without looking at detailed logs, an error may occur if there is a conflict between HTTP connections.
Why not try explicitly closing the HTTP connection with "con.close()"?
I thought that closing the HTTP connection at the end of processing would reduce the possibility of resource contention even if the environment was reused.

profile picture
專家
已回答 1 個月前
profile picture
專家
已審閱 1 個月前
  • I appreciate the quick response. No there isn't much detail to the logging. Yes, I had explicit close before putting in the except. I was hoping to get more detail printing the exception, still trying to figure out how to get debug level logs.

    My best guess was the <botocore.response.StreamingBody> resource. So, I tried reading the stream into memory ~200MB ("ts" media archive file), close the StreamBody and post the Byte array, but got the same thing.

    That led me to believe it was the client not handling the stream, the stream pointer issue, need to chunk the reading of the stream so I tried to use the more current requests and urllib3 clients, but they are not available and I don't want to go through the layer process to find out that is not the issue. I would rather convert from Python to Java to see if I can get information on what the "Device" is referring to. That an I am a once in a few years Python coder so I am not overly familiar with the Python libraries.

    I ruled out a network issue since I have another java microservice posting JSON to the same gateway and I didn't get an http error.

    I just thought it would be a common easy thing to read s3 object and post using Python.

  • Basically it came down to the client post. I have abandoned the Python version and created a Spring Function "Function<S3Event, Void>" that successfully posted the video TS file without reading the stream into memory.

您尚未登入。 登入 去張貼答案。

一個好的回答可以清楚地回答問題並提供建設性的意見回饋,同時有助於提問者的專業成長。

回答問題指南