How to resolve “Client.InvalidKMSKey.InvalidState” error when creating an ElasticBeanstalk environment ?

This issue usually occurs when you have enabled EBS volume automatic encryption [1] using a customer managed KMS key. EBS volumes can be automatically encrypted from the EC2 console > Settings > Data protection and security > Encryption. Once enabled, customer can choose to encrypt their volumes with either AWS managed KMS key or using KMS key managed by customer. Using AWS managed KMS keys, the key already have the required permission for Auto Scaling.

However, using a customer managed key, If the KMS key policy does not have a policy statement allowing the role “AWSServiceRoleForAutoScaling” to use the key, the environment creation will fail with the error “Client.InvalidKMSKey.InvalidState”. In the environment events, you will see an error similar to the following:

“Creating Auto Scaling group named: XXXXXX failed. Reason: Resource handler returned message: "Group did not stabilize. Last scaling activity: Instance became unhealthy while waiting for instance to be in InService state. Termination Reason: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state""

In CloudTrail, you can see that the API calls made by the service role “AWSServiceRoleForAutoScaling” to use the customer managed KMS key failed with AccessDenied. For example:

"eventSource": "kms.amazonaws.com", 
"eventName": "GenerateDataKeyWithoutPlaintext", 
"awsRegion": "eu-west-1", 
"sourceIPAddress": "autoscaling.amazonaws.com", 
"userAgent": "autoscaling.amazonaws.com", 
"errorCode": "AccessDenied", 
"errorMessage": "User: arn:aws:sts::<account-ID>:assumed-role/AWSServiceRoleForAutoScaling/AutoScaling is not authorized to perform: kms:GenerateDataKeyWithoutPlaintext on resource: arn:aws:kms:eu-west-1:<account-ID>:key/<Key-ID> because no identity-based policy allows the kms:GenerateDataKeyWithoutPlaintext action"

In the above API call, the user calling the action is AWS Auto Scaling “User: arn:aws:sts::<account-ID>:assumed-role/AWSServiceRoleForAutoScaling/AutoScaling”. For example, the user identity in CloudTrail:

"userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAXZFJXT3ZUS6JUAFKE:AutoScaling",
    "arn": "arn:aws:sts::111122223333:assumed-role/AWSServiceRoleForAutoScaling/AutoScaling",
    "accountId": "111122223333",

The role “AWSServiceRoleForAutoScaling” is a managed service role used by EC2 Auto Scaling for all scaling activities in the account. Therefore, any action performed by this role to a resource that uses the same KMS key may be subject to permission errors.

Resolution

For customer managed keys, the role “AWSServiceRoleForAutoScaling" needs to be included in the principal statement in the key policy [2] to allow Auto Scaling activities to use the key.

To resolve the issue, include the following statement in your KMS key policy:

{
"Sid": "Allow service-linked role use of the customer managed key", 
"Effect": "Allow", 
"Principal": {
"AWS": [ 
"arn:aws:iam::account-id:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling" ] 
}, 
"Action": [ 
"kms:Encrypt", 
"kms:Decrypt", 
"kms:ReEncrypt*", 
"kms:GenerateDataKey*", 
"kms:DescribeKey" ], 
"Resource": "*" 
}

The above statement will allow the service-linked role named “AWSServiceRoleForAutoScaling” used by Auto Scaling to use the customer managed key for scaling operations.

Once updated, you will be able to create a new ElasticBeanstalk environment successfully.

Note: The previous environment will be in unrecoverable state [3], therefore, you won’t be able to make any updates on it as the CloudFormation stack will be in CREATE_FAILED status.

References: [1] https://repost.aws/knowledge-center/ebs-automatic-encryption [2] https://docs.aws.amazon.com/autoscaling/ec2/userguide/key-policy-requirements-EBS-encryption.html#policy-example-cmk-access [3] https://repost.aws/knowledge-center/elastic-beanstalk-invalid-state