AWS KMS: Receiving a ThrottlingExceptions error when making requests to AWS KMS

Question

Hi Team,
Since a couple of months, we have been facing issues with throttling on AWS KMS which directly impacts our production application too.
On further investigation we have seen that cloudwatch logs are generating a large number of requests on AWS KMS (more than 50,000 requests per day). As a temporary solution we have deactivated kms encryption on cloudwatch logs related to non-prod environments. Now the current AWS KMS requests has come down to around 33000 per day.
Request some guidance from your side on mitigating this issue.

Answer

Unfortunately there are cases where KMS api calls are just to frequent in production- or load-test loads. You'll hit the quota of the KMS APIs and get throttled.

There are 4 ways to avoid being throttled:
1. change the quota/limit: Some of the quota's can be increased by filing a support ticket to AWS.
2. use data key caching: If you are in control of the use of the AWS Encryption SDK (since you log the Throttling error you might be able to use this method)
3. use different kind of encryption at rest. (Cloudwatch supports encryption at rest by default AWS managed SSE keys are used that will not get throttled)
4. make less log lines, a good practice is to log one json object that contains everything for one request

AWS KMS: Receiving a ThrottlingExceptions error when making requests to AWS KMS

Relevant content