Skip to content

Issue with Creating Spot Fleet Using Encrypted AMI from Another Account

0

I was attempting to create a Spot Fleet using an AMI with an encrypted volume from another AWS account (Account-A). We created an AMI (ami-112234) with an encrypted volume in that account using their KMS key (arn:aws:kms:eu-west-1:Account-A:key/121212121). Both the AMI and KMS key have been shared with all accounts within our organization, and we've granted all required permissions like that

{
	"Version": "2012-10-17",
	"Id": "key-consolepolicy-3",
	"Statement": [
		{
			"Sid": "Allow use of the key",
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"arn:aws:iam::ACCOUNT-B:root"
				]
			},
			"Action": [
				"kms:Encrypt",
				"kms:Decrypt",
				"kms:ReEncrypt*",
				"kms:GenerateDataKey*",
				"kms:DescribeKey",
				"kms:CreateGrant"
			],
			"Resource": "*"
		},
		{
			"Sid": "Allow EC2 Spot Fleet",
			"Effect": "Allow",
			"Principal": {
				"AWS": [
					"arn:aws:iam::Account-B:role/aws-service-role/spotfleet.amazonaws.com/AWSServiceRoleForEC2SpotFleet",
					"arn:aws:iam::Account-B:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling",
					"arn:aws:iam::Account-B:role/my-ec2-spot-fleet-instance-service-iam-role"
				]
			},
			"Action": [
				"kms:Decrypt",
				"kms:DescribeKey",
				"kms:ReEncrypt*",
				"kms:Encrypt",
				"kms:GenerateDataKey*",
				"kms:CreateGrant"
			],
			"Resource": "*"
		},
		{
			"Sid": "AllowEC2ServiceToUseKey",
			"Effect": "Allow",
			"Principal": {
				"Service": "ec2.amazonaws.com"
			},
			"Action": [
				"kms:Decrypt",
				"kms:DescribeKey",
				"kms:ReEncrypt*",
				"kms:Encrypt",
				"kms:GenerateDataKey*",
				"kms:CreateGrant"
			],
			"Resource": "*"
		}
	]
}

I tried to create the Spot Fleet using this IAM role: my-ec2-spot-fleet-instance-service-iam-role, with all necessary permissions added. However, when attempting to create the Spot Fleet, I received error messages as shown in the attachment.

{
  "Version": "2012-10-17",
  "Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "ssm:GetParameters",
            "ssm:GetParameter",
            "ssm:GetParametersByPath",
            "kms:Decrypt",
            "kms:ReEncrypt*",
            "kms:Encrypt",
            "kms:GenerateDataKey*",
            "kms:DescribeKey",
            "kms:CreateGrant",
            "kms:GenerateDataKey*"
       ],
        "Resource": ["*"]
    },
    {
        "Effect": "Allow",
        "Action": [
            "ec2:RunInstances",
            "ec2:CreateTags",
            "ec2:RequestSpotFleet",
            "ec2:ModifySpotFleetRequest",
            "ec2:CancelSpotFleetRequests",
            "ec2:DescribeSpotFleetRequests",
            "ec2:DescribeSpotFleetInstances",
            "ec2:DescribeSpotFleetRequestHistory",
            "ec2:DescribeSubnets",
            "ec2:TerminateInstances",
            "ec2:DescribeImages",
            "ec2:DescribeInstanceStatus",
            "ec2:RequestSpotInstances",
            "ec2:AttachVolume",
            "ec2:DetachVolume",
            "ec2:ModifyInstanceAttribute",
            "ec2:DescribeVolumes",
            "ec2:DescribeInstances"
        ],
        "Resource": "*"
    }
  ]
}

Interestingly, I was able to successfully create an EC2 instance using the same AMI. I noticed that when I attempted to create a Spot Fleet, an EBS volume with the aforementioned KMS key was attached. However, when creating a regular EC2 instance, that volume was not attached.

Cause: One or more of the attached Amazon EBS volumes are encrypted with an inaccessible AWS KMS key. Resolution: • Ensure that the KMS kevs are in the enabled state • Ensure that you have the following permissions to decrypt and encrypt volumes: • "kms:CreateGrant" • "kms:Decrypt" • "kms:DescribeKey" • "kms:GenerateDataKeyWithoutPlain Text" • "kms:ReEncrypt" • If the instance was launched on your behalf by another AWS service, such as Amazon EC2 Auto Scaling, ensure that the KMS key policies grant that service access to the KMS key.

After researching, I found information suggesting that shared AMIs with encrypted volumes cannot be used directly across accounts and must be copied from the source to the destination account. Is this correct, or am I missing something in my configuration?

Reference: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html

4 Answers
0

When working with encrypted AMIs across accounts for Spot Fleet, there are specific permission requirements that need to be addressed.

The issue you're experiencing is related to how the Spot Fleet service accesses the KMS key used to encrypt the AMI. While your policy includes many necessary permissions, there are a few key elements missing.

For encrypted AMIs, when using a Spot Fleet, you need to ensure that the AWSServiceRoleForEC2SpotFleet service-linked role has proper permissions to use the customer managed key (CMK). Your current policy is missing some required permissions and has structural issues.

The key policy should include two specific statement blocks:

  1. "Allow use of the key" statement that includes permissions for:
  • kms:Encrypt
  • kms:Decrypt
  • kms:ReEncrypt*
  • kms:GenerateDataKey*
  • kms:DescribeKey
  1. "Allow attachment of persistent resources" statement that includes:
  • kms:CreateGrant
  • kms:ListGrants
  • kms:RevokeGrant

Your current policy is missing the kms:ListGrants and kms:RevokeGrant permissions, which are essential for the Spot Fleet service to work with encrypted AMIs across accounts.

Additionally, when using grants (which is the recommended approach for cross-account scenarios with AWS services), you should include a condition that restricts grant creation to AWS resources:

"Condition": {"Bool": {"kms:GrantIsForAWSResource": true}}

While it's true that you can launch a regular EC2 instance with the shared encrypted AMI, Spot Fleet has additional requirements because it's a service that launches instances on your behalf, requiring more extensive KMS permissions.

Regarding your question about copying AMIs: While copying an encrypted AMI to your account is one solution (and gives you full control), it's not strictly required. You can use a shared encrypted AMI directly if the proper KMS key permissions are configured correctly for the service roles involved.
Sources
Spot Fleet permissions - Amazon Elastic Compute Cloud
Troubleshoot copy errors between Regions or Accounts in Amazon EC2 | AWS re:Post
ASG KMS customer managed key | AWS re:Post

answered a year ago

0

You can share AMIs encrypted with customer-managed KMS keys across accounts and launch EC2 instances directly from them. However, there are some important considerations:

  1. For the target account (Account-B in your case), the IAM user or role needs specific permissions to perform these KMS operations on the source account's KMS key:
  • kms:DescribeKey
  • kms:ReEncrypt*
  • kms:CreateGrant
  • kms:Decrypt
  1. While launching instances from a shared encrypted AMI, AWS recommends re-encrypting the volumes using a KMS key in the target account rather than using the source account's KMS key. This protects you if:
  • The source KMS key is compromised
  • The source account revokes permissions
  • You lose access to encrypted volumes created using the source KMS key

The error you're experiencing with Spot Fleet might be related to the re-encryption process. When launching a regular EC2 instance, you have the option to specify a different KMS key for re-encryption in the target account. Try modifying your Spot Fleet configuration to include re-encryption settings using a KMS key from your account (Account-B).

You don't need to copy the AMI to your account first - sharing encrypted AMIs across accounts is supported with a single API call, as long as the proper KMS key permissions are in place.

The key difference between your EC2 instance success and Spot Fleet failure might be related to the re-encryption settings rather than the sharing mechanism itself.

References

[1] How to share encrypted AMIs across accounts to launch encrypted EC2 instances

[2] Share an AMI with specific AWS accounts

AWS
EXPERT

answered a year ago

0

Hi, Thanks for your answer. I've already added every permission that mentioned in AWS documentation and your answers as well. Unfortunately, it didn't work. When I run the following command, I received **"implicitDeny" **and "AllowedByOrganizations": false message. Can I assume that it has been restricted by SCP?

aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::{TARGET-ID}:role/my-ec2-spot-fleet-instance-service-iam-role \
  --action-names kms:DescribeKey \
  --resource-arns arn:aws:kms:eu-west-1:{SOURCE-ID}:key/121232-cecee343-3eecba8985b9

I got that

{
    "EvaluationResults": [
        {
            "EvalActionName": "kms:DescribeKey",
            "EvalResourceName": "arn:aws:kms:eu-west-1:{SOURCE-ID}:key/121232-cecee343-3eecba8985b9",
            "EvalDecision": "implicitDeny",
            "MatchedStatements": [],
            "MissingContextValues": [],
            "OrganizationsDecisionDetail": {
                "AllowedByOrganizations": false
            }
        }
    ]
}

answered a year ago

0

Hi,

I was working surprisingly when on call with AWS Support without making any changes :D

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.