Lambda has errors without invocations

It sounds like your Lambda function is configured to be triggered by an S3 event (object creation), but the errors you're seeing without invocations in CloudWatch metrics can be caused by several factors. Here's a step-by-step guide to help you troubleshoot and pinpoint the issue:

1. Check Lambda’s Invocation Permissions: Ensure that the S3 bucket has the proper permissions to invoke the Lambda function. The S3 bucket needs to have an invoke permission set up correctly in the Lambda’s resource policy.

You can verify this by checking the Lambda function’s policy to make sure S3 is authorized to trigger it.

Lambda policy example:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "lambda:InvokeFunction", "Resource": "arn:aws:lambda:REGION:ACCOUNT_ID:function:YOUR_LAMBDA_FUNCTION", "Principal": { "Service": "s3.amazonaws.com" }, "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:::YOUR_BUCKET_NAME" } } } ] } 2. Check Lambda Event Source Mapping (S3 Notifications): Ensure that the S3 bucket notification is properly configured to trigger the Lambda function. You can check this by going into the S3 console and reviewing the notifications for the bucket.

S3 notifications may not trigger Lambda for every upload if there is any misconfiguration in the event notification. Verify that the event type (e.g., s3:ObjectCreated:*) is correctly set up to trigger on the specific event you're interested in.

3. Investigate Lambda Metrics and Logs: Check the CloudWatch metrics for Lambda invocations to ensure there isn't a misconfiguration that is causing the errors. It is possible that the errors are being logged due to some invocation retries or other conditions not directly related to the file upload.

The fact that there are errors without invocations indicates that there may be an issue before the actual Lambda invocation (e.g., S3 notification issues, AWS service glitches, or Lambda initialization errors). Check the Lambda logs around the time when the file was uploaded to see if there is any additional context that explains the errors.

4. Consider Event Throttling: Lambda has concurrency limits, and if you exceed the available capacity or if there are too many notifications coming in a short period, it can result in throttling. Check if there were multiple uploads close to the same time or if the S3 notifications were sent in bursts that could overwhelm the Lambda function.

Look for throttling-related logs in CloudWatch or check the AWS Lambda Throttling metrics to see if the errors coincide with high invocation rates.

5. Check S3 Event Configuration for Failed Events: S3 may also fail to trigger Lambda if there is an issue with the S3 event itself (e.g., malformed or invalid event structure). When this happens, Lambda won't be invoked but may still show errors due to some internal retries or other mechanisms in the AWS platform.

Check S3 notification settings to make sure the event is set correctly, and that the Lambda function is being targeted properly.

6. Examine Lambda Execution Role: If your Lambda function requires access to other AWS resources (e.g., reading from S3), ensure that the Lambda execution role has the necessary permissions. If the Lambda doesn’t have the proper permissions to interact with the S3 bucket or other services, the invocation could fail even though the event is triggered.

Lambda role policy example for S3 access:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::YOUR_BUCKET_NAME", "arn:aws:s3:::YOUR_BUCKET_NAME/*" ] } ] } 7. Investigate S3 Notification Retries: When S3 sends an event notification to Lambda, it will retry the notification if the Lambda invocation fails (due to errors, permissions, etc.). This can result in situations where S3 successfully retries but the Lambda fails due to a configuration issue.

Check for any automatic retries or failed attempts in the Lambda logs. This can help identify if the initial failures were part of a retry process.

8. Look at Lambda Cold Starts or Timeouts: A cold start might be delaying the Lambda invocation, especially if the Lambda takes longer to initialize. While this might not be the root cause, it could contribute to delays that might make it appear that there are errors in the Lambda metrics.

Make sure that your Lambda has sufficient memory and timeout settings to handle large events or heavy loads.

9. Look for Dead Letter Queue (DLQ) Configurations: If you’ve configured a Dead Letter Queue (DLQ) for your Lambda function, check the DLQ for any messages related to the errors. This can help you pinpoint any issues with failed invocations.

Lambda DLQ: If you haven’t set up a DLQ yet, consider configuring it to capture errors in case invocations fail.

10. Monitor S3 Events via CloudTrail: You can monitor the S3 event notifications by reviewing AWS CloudTrail logs to see when S3 is sending notifications to Lambda. This can help you see if notifications are being sent as expected and if there are any issues at the S3 notification level.

regards, M Zubair https://zeonedge.com

Thanks for the detailed answer! We haven't have the same issues with errors without invocations since and the lambda is usually running fine so this might remain a mystery for now

1. We checked that we have the correct policy on the lambda (and it is usually correctly triggered by the S3 events)
2. We have the correct event configured to trigger the notification
3. The only logs in CloudWatch for that time period were for the successful execution
4. We initially suspected throttling but we don't have any throttling events for that time period
5. The S3 event looks like it's been set up correctly (and it has been working since)
6. The lambda IAM role looks correct and has been working since
7. We didn't see any trace of the error calls without invocation in the lambda logs
8. We didn't get any timeout, from the monitoring and the logs it looks like the lambda was not invoked at all for the failed events (and for successful invocations we are well under the limit)
9. We had not set up a DLQ, we've added one now in case it happens again
10. We haven't turned CloudTrail on, but thank you for the suggestion and we will do if this issue reccurrs

Thanks for taking the time to try and help us M Zubair