Help us improve the AWS re:Post Knowledge Center by sharing your feedback in a brief survey. Your input can influence how we create and update our content to better support your AWS journey.
Lambda with response stream showing an error in metric, but nothing in logs
Hi,
I've moved on of our old node services from Elastic Beanstalk to a Lambda with a function URL returning a response stream. The service that was ported weas only on EB as it's response can be more than 6MB and we use the Serverless framework with Node for our lambdas. This lambda is using Node 14.x.
Everything works find most of the time, but occasionally the Error count and success rate (%) metric shows an error but looking that the logs the lambda completed successfully.
I have turned x-ray on to see if that showed anything, but it shows all the trace status where ok, and everything looks ok when you drill down into a trace.
Does anyone have any pointers where I could look to see what is going on?
- Newest
- Most votes
- Most comments
Try to catch the exception and log the exception with level 'ERROR' to view in log group.
Refer below link for logging using nodejs
https://docs.aws.amazon.com/lambda/latest/dg/nodejs-logging.html
I'm facing the same issue. I have a Docker image deployed to Lambda with a function URL with the RESPONSE_STREAM invoke mode. The function's monitoring shows errors while CloudWatch logs, X-Ray traces, and Lambda Insights don't show any errors.
When I switch the function URL's invoke mode to BUFFERED and change the function's code to work in the buffered mode, all errors disappear.
Thanks, but we need to use the RESPONSE_STREAM as the response can be larger than 6MB.
Hey, I'm facing the same error, and I also need to use RESPONSE_STREAM. In my case, the execution is working 100% (so there is no problems with logs). However, it keeps showing that the execution ends with a timeout. Did you guys find any way to deal with it?
Sorry we haven't found a solution, and the service is in the process of being retried.
I'm seeing this too. I'm using a Lambda with RESPONSE_STREAM's to stream out LLM responses, and things are working fine, but now that its getting hit a bit more these errors in CloudWatch are unsettling. Error rate is about 0.05-0.1.
I've jacked SYSTEM and APP level logging on my lambda, see more errors happen in the charts, but nothing in the logs. Memory looks good, duration is fine, yet these errors.
The only thing i can think of is given this is not behind APIGW or cloudwatch and i wonder if clients kill connections prematurely or something like that it goes outside the realm of the logs and just into the void.
Would love a response from AWS on this, it selfishly feels like a bug with Lambda streaming.
One solution that we've found helps is ensuring all work is completed before closing the response stream. We were previously closing the response stream then awaiting any side effect work which was causing this problem of intermittent CloudWatch errors. I haven't been able to find any documentation on this, but it seems like the Lambda is being shut-down once the response stream is closed.
Relevant content
- AWS OFFICIALUpdated 10 months ago
Thanks. I've already done that using the Winston logger, but I'll give it a try with console. error just to make sure nothing is being missed.