KMS error when deploying new infrastructure

Question

My configuration is: AWS - Gitlab - Terraform.

Infrastructure being deployed on gitlab using AWS SECRET KEY and ID.

Key described above are linked to a user that have the following policies:
	
AdministratorAccess	AWS managed - job function	Provides full access to AWS services and resources.	10 years ago

AmazonEC2FullAccess  AWS managed	Provides full access to Amazon EC2 via the AWS Management Console.	6 years ago

AmazonEKSClusterPolicy	AWS managed	This policy provides Kubernetes the permissions it requires to manage resources on your behalf. Kubernetes requires Ec2:CreateTags permissions to place identifying information on EC2 resources including but not limited to Instances, Security Groups, and Elastic Network Interfaces.	19 days ago

AmazonEKSServicePolicy	AWS managed	This policy allows Amazon Elastic Container Service for Kubernetes to create and manage the necessary resources to operate EKS Clusters.	6 months ago

DevelopersPolicy	Customer inline deployer-gl-integration-policy-ft-managed	Customer managed	the deploy policy	4 years ago

IAMFullAccess	AWS managed	Provides full access to IAM via the AWS Management Console.	5 years ago

terraform-account-init	Customer managed	Init the account	2 years ago

kms-manual-key-policy-hp	Customer managed	this key enables KMS key access, created by Hardik Prajapati manually	2 hours ago

ec2-kms-iam-policy-hp	Customer managed	Manually created key to resolve KMS issue	1 hour ago

ec2-kms-iam-policy-hp and kms-manual-key-policy-hp where added after the issue.

Gitlab is self hosted.
The error on gitlab is :

```
	│ Error: waiting for EC2 Instance (i-030874ad7943b829a) create: unexpected state 'shutting-down', wanted target 'running'. last error: Client.InvalidKMSKey.InvalidState: The KMS key provided is in an incorrect state
	│ 
	│   with aws_instance.engineering-windows-docker-runner,
	│   on main.tf line 9, in resource "aws_instance" "engineering-windows-docker-runner":
	│    9: resource "aws_instance" "engineering-windows-docker-runner" {
```

On AWS we are getting this error:

```
	Instance failed to launch
	Cause: One or more of the attached Amazon EBS volumes are encrypted with an inaccessible AWS KMS key.

Resolution:
	Ensure that the KMS keys are in the enabled state.
	Ensure that you have the following permissions to decrypt and encrypt volumes:
	"kms:CreateGrant"
	"kms:Decrypt"
	"kms:DescribeKey"
	"kms:GenerateDataKeyWithoutPT"
	"kms:ReEncrypt"
	If the instance was launched on your behalf by another AWS service, such as Amazon EC2 Auto Scaling, ensure that the KMS key policies grant that service access to the KMS key.
For more information, see Instance stops or terminates immediately
```

On link https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-launch.html#troubleshooting-launch-internal
It is suggested to add two specific rights, which were added too.

The issue arrived after some manipulations of the kms key (to test Kubernetes implementations EKS). Normally unrelated to the current issue.
A user managed key provisioned by terraform was manually dissabled and scheduled for deletion, this was reverted after the error was noticed.

The procedures were working for years already so we have depleted the initial ideas we had. Impacted services are related to autoscaling proper to gitlab, AWS deployment of individual instances so no technologiy proper to gitlab, plus some CLI related command like starting instances.

Best regards,
Miguel

re:Post Agent · Answer

It appears that you're experiencing an issue with KMS (Key Management Service) when deploying new infrastructure using Terraform through GitLab CI/CD. The error suggests that the KMS key used for encrypting EBS volumes is in an incorrect state or inaccessible.

Here are some steps to troubleshoot and potentially resolve the issue:

1. Verify KMS key state: Ensure that the KMS key used for encrypting EBS volumes is in the "Enabled" state. If it was previously disabled or scheduled for deletion, make sure it has been fully re-enabled.

2. Check KMS key permissions: Confirm that the IAM user or role used by Terraform has the necessary permissions to use the KMS key. The required permissions include:
   - kms:CreateGrant
   - kms:Decrypt
   - kms:DescribeKey
   - kms:GenerateDataKeyWithoutPlaintext
   - kms:ReEncrypt

3. Review KMS key policy: If the instance is being launched by another AWS service (like EC2 Auto Scaling), ensure that the KMS key policy grants access to that service.

4. Verify Terraform configuration: Check your Terraform configuration to ensure it's using the correct KMS key ARN for EBS volume encryption.

5. IAM role permissions: If you're using an IAM role for EC2 instances, make sure it has the necessary permissions to access the KMS key.

6. Clean up Terraform state: If the issue persists, try running `terraform refresh` to update the state, followed by `terraform plan` and `terraform apply`. This can help resolve any discrepancies between the actual infrastructure and Terraform's state.

7. Check AWS CloudTrail: Review CloudTrail logs for any KMS-related errors or access denied events that might provide more insight into the issue.

8. Temporary workaround: As a temporary measure, you could try launching the instance without encryption and then add encryption later, although this is not recommended for production environments.

If the issue continues after trying these steps, you may need to reach out to AWS support for further assistance, especially if you suspect there might be an issue with the KMS service itself.  
**Sources**  
[Deploying Containers on AWS: A Guide to ECS and EKS | AWS re:Post](https://repost.aws/articles/ARMYRTVoV1RKSs-0Zy1S2LgA/deploying-containers-on-aws-a-guide-to-ecs-and-eks)  
[Connect your on-premises Kubernetes cluster to AWS APIs using IAM Roles Anywhere | AWS Security Blog](https://aws.amazon.com/blogs/security/connect-your-on-premises-kubernetes-cluster-to-aws-apis-using-iam-roles-anywhere/)

Shahad_C · Answer

It sounds like the AutoScaling Group (ASG) being used for the cluster isn't able to access a CMK (Customer Managed KMS Key).  You'll need to allow the Service Linked Role (SLR) of the ASG in the Key Policy, and then in the ASG account call the create-grant CLI command.  After doing *both* of these, it should work.  More info:
https://docs.aws.amazon.com/autoscaling/ec2/userguide/key-policy-requirements-EBS-encryption.html#policy-example-cmk-cross-account-access

KMS error when deploying new infrastructure

Relevant content