Lambda too slow for sending sqs messages

Question

Hello,

currently we are using a lambda which is triggered by our s3 bucket and goes through each line of a file and sends it via SQS to another lambda to process that data.

Our issue is, that it takes to long. There could be up to millions of data which needs to be send via SQS,
but it takes too long to send the sqs messages. We could use batching here but I think it wont help a lot because each data is about 40KB already.

I tried:

- increasing RAM of lambda

- running concurrent promises which sends SQS messages

Both approaches did increase the speed but not enough. (We can now process about 500k and then run into a 15min timeout by the lambda)

Is there any other way to increase the speed with aws lambda? or somehow pick up the line where the lambda timed out?

Uri · Answer

I would recommend a different approach. Instead of invoking a Lambda function when you upload a file to S3 and sending messages to SQS, start a Step Functions execution (send the S3 event to EventBridge and set the target to be Step Functions). The state machine will use a Distributed Map state to iterate over the lines in the file (if they are CSV or JSON), and will invoke a Lambda function to process the lines. You can use batches so that the Lambda function will be invoked on a batch of lines.

Lambda too slow for sending sqs messages

Relevant content