By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Return Value from Lambda function triggered by SQS to individual client

0

Hi all, I'm trying to implement a simple webservice which processes uploaded files and returns a response to the uploader. Based on AWS I've got this (incomplete) workflow in my mind:

  1. Client uploads file to S3.
  2. Client sends SQS message which triggers the Lambda function.
  3. Lambda downloads and processes the file and uploads the result to S3.
  4. As soon as processing is finished the individual uploader is notified and downloads the result from S3.

Now I'm facing the following problem: I'd like to benefit from the advantages of combing AWS Lambda with SQS. On the other hand, I lose the capability to directly return sth from the Lambda function to the client. How could this loss be replaced? I've already had SNS in mind, which seems rather suitable for publishing notifications to a whole bunch of (less dynamic) recipients instead of one individual client per Lambda call. Is EventBridge with AWS Gateway as target maybe the right direction? I again face the problem of filtering out the correct recipient for an individual "AWS Lambda response event". Or is it the only solution to drop the SQS service which then allows to directly send Lambda output to the client?

Remarks: Input file size: 1-25MB Lambda processing time: ~20sec

thanks for any advice!

Topics
ServerlessComputeFront-End Web & MobileNetworking & Content DeliveryApplication Integration
Tags
AWS LambdaAmazon API GatewayAmazon EventBridgeAmazon Simple Queue ServiceAmazon Simple Notification Service (SNS)
Language
English
Neonglb
asked 2 years ago1743 views
1 Answer
2
Accepted Answer

You didn't mention what is the client? Is it a browser/mobile device? Or is another server component? How is the event added to SQS?

If it is a browser/mobile device you could use a WebSocket from the client to API Gateway, include the connection ID in the SQS message, and when the Lambda function is done, it will send a notification on that connection. If it is a mobile device, you could send a mobile push notification (you can send to a single device, you do not need to broadcast it to all users). Lastly, you can make the call synchronous, i.e., client calls API GW, that invoke the Lambda function that returns the value. Note that the last option may have scaling issues, depending on the number of requests.

If it is a server component, you could use EventBridge with each server component creating a rule with its own service ID, which should be included in the original SQS message. Another option is for each such service to create an incoming queue, to which the Lambda function will send the completion event.

I guess there are other options, but they also depend on your exact requirements.

profile pictureAWS
EXPERT
Uri
answered 2 years ago
profile picture
EXPERT
Sedat Salman
reviewed 10 months ago
  • Neonglb
    2 years ago

    Thanks a lot for your fast response. Sorry, the client is a Software running on Windows with many individual users. The websocket approach sounds great! Is it recommended then to exclusively communicate through the websocket connection (trigger SQS message to Lambda, sending response from lambda, maybe even S3 upload/result download?) Or is executing each step via AWS SDK more clean (upload to s3, then open websocket connection, afterwards send SQS message which contains connection id, etc.)? And if a file is already uploaded before calling the Lambda, how can I avoid that it's the client's responsibility to chose a valid S3 object identifier (which does not yet exist) and to provide it to the lambda invocation?

  • Uri EXPERT
    2 years ago

    If it is another server component, I would probably use the SQS approach, i.e., create an incoming queue for each such Windows server that will be used by the Lambda function that processes the files to send back the response. It will eliminate the need for additional components such as API Gateway. I would let the "client" create the file in S3, send the object key along side the incoming queue URL to the SQS used by the processing function. The processing function will get messages from that queue, process the object and send the response.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content