How do I use CloudTrail LookupEvents to find the source of "Rate exceeded" errors associated with Lambda function management event API calls?

Resolution

You might receive a "Rate exceeded" error when your environment exceeds Lambda function quota limits for API requests. Before you resolve a "Rate exceeded" error, review your CloudTrail event logs.

1.    Run the CloudTrail lookup-events AWS Command Line Interface (AWS CLI) command to view the total count of management event API calls.

Note: Set the start-time, end-time, and time zone for each value to match the problematic timeframe.

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=lambda.amazonaws.com --start-time YYYY-MM-DDTHH:MM:SS+00:00 --end-time YYYY-MM-DDTHH:MM:SS+00:00 | grep -i EventName | grep -v CloudTrailEvent | sort | uniq -c | sort -r

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

Example CloudTrail lookup-events AWS CLI command:

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=lambda.amazonaws.com --start-time 2023-04-03T18:29:00+05:30 --end-time 2023-04-03T18:29:30+05:30 | grep -i EventName | grep -v CloudTrailEvent | sort | uniq -c | sort -r

Example CloudTrail lookup-events AWS CLI command output:

2072    "EventName": "ListVersionsByFunction20150331",
 877    "EventName": "GetFunction20150331v2",
 245    "EventName": "GetFunctionCodeSigningConfig",
 210    "EventName": "UpdateFunctionCode20150331v2",
 210    "EventName": "ListTags20170331",
 206    "EventName": "GetRuntimeManagementConfig",
 121    "EventName": "PublishVersion20150331",
 106    "EventName": "UpdateAlias20150331",
 89     "EventName": "GetPolicy20150331v2",
  2     "EventName": "ListLayers20181031",

2.    Run the CloudTrail lookup-events command to view the total count of API calls that each user made.

Note: Set the start-time, end-time, and time zone for each value to match the problematic timeframe.

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=lambda.amazonaws.com --start-time YYYY-MM-DDTHH:MM:SS+00:00 --end-time YYYY-MM-DDTHH:MM:SS+00:00 | grep -i Username | grep -v CloudTrailEvent | sort | uniq -c |. sort -r

Example CloudTrail lookup-events AWS CLI command:

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventSource,AttributeValue=lambda.amazonaws.com --start-time 2023-04-03T18:29:00+05:30 --end-time 2023-04-03T18:29:30+05:30 | grep -i Username | grep -v CloudTrailEvent | sort | uniq -c | sort -r

Example CloudTrail lookup-events AWS CLI command output:

4047   "Username": "AWSCloudFormation",
 89    "Username": "auditor",
  2    "Username": "appsync",

3.    If necessary, run the CloudTrail lookup-events command again to identify users or events that might be making a high number of API calls.

4.    After you review the output of the CloudTrail lookup-events command, review the source of the API calls that exceed the Lambda function quota limits.

Related information

How do I prevent "Rate exceeded" errors in CloudFormation?

How do I troubleshoot Lambda function throttling with "Rate exceeded" and 429 "TooManyRequestsException" errors?