How do I troubleshoot a FAILED Image Builder lifecycle policy, or a policy that completed but still has images available?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshooting errors for the AWS CLI. Also, make sure that you're using the most recent AWS CLI version.

Troubleshoot a lifecycle policy that has a FAILED status

If a lifecycle policy has a FAILED status, then run the following get-lifecycle-execution AWS CLI command to check why:

aws imagebuilder get-lifecycle-execution --example-lifecycle-execution-id --region example-region

Note: Replace example-lifecycle-execution-id with the execution ID of the lifecycle policy and example-region with your AWS Region.

Example output:

{        "state": {
            "status": "FAILED",
            "reason": "Lifecycle execution failed. Reason: One or more resources failed."
        }

To allow Image Builder to run lifecycle policies, you must use the default EC2ImageBuilderLifecycleExecutionRole role. Or, create a custom AWS Identity and Access Management (IAM) role for Image Builder lifecycle management. If you use a role that doesn't have the required configuration, then the lifecycle policy fails.

If you use an Amazon Elastic Container Registry (Amazon ECR) container image, then you must add the LifecycleExecutionAccess: EC2 Image Builder tag to your Amazon ECR repository. If you don't, then you receive the following Reason in the get-lifecycle-execution command output:

"ImageBuilder is not authorized to perform: ecr:BatchGetImage on resource"

To tag the Amazon ECR container image, see Tag a resource from the AWS CLI.

Troubleshoot issues where the lifecycle policy completed, but images are still available

Even when the lifecycle policy completes, there might still be Amazon Machine Images (AMIs), Amazon Elastic Block Store (Amazon EBS) snapshots, or container images available.

To troubleshoot this issue, run the following get-lifecycle-policy command to check policy details:

aws imagebuilder get-lifecycle-policy --lifecycle-policy-arn example-policy-arn --region example-region

Note: Replace example-policy-arn with the lifecycle policy's Amazon Resource Name (ARN) and example-region with your Region.

Then, check the following configurations.

Check the lifecycle policy rule conditions

You must configure Image Builder to delete resources after the lifecycle policy completes. If you don't, then Image Builder keeps the images.

To check the resources that the lifecycle policy deletes, check the policyDetails section of the get-lifecycle-policy command output:

"policyDetails": [
            {
                "action": {
                    "type": "DELETE",
                    "includeResources": {
                        "amis": false,
                        "snapshots": false
                    }

Check the IncludeResources field to identify the AMIs, snapshots, and container images that the lifecycle policy marked for deletion or retention. If they're marked as false then Image Builder retains the resource after it runs the lifecycle policy.

To update the resources that Image Builder deletes, update your lifecycle policy, and then run the following update-lifecycle-policy command:

aws imagebuilder update-lifecycle-policy --lifecycle-policy-arn example-policy-arn 

Note: Replace example-policy-arn with the ARN of the updated lifecycle policy.

Or, complete the following steps to use the Image Builder console to update your lifecycle policy:

1. Open the Image Builder console.
2. Choose Lifecycle policies.
3. Select your policy.
4. Choose Rules, and then select Rule conditions.
5. For Image Builder to delete all resources after the lifecycle policy completes, select Deregister associated AMIs and delete their snapshots and Delete associated ECR container images.

Check whether the lifecycle policy rules retain images

For the Delete rules lifecycle policy, you can select By age or By count to retain images for recipes that the rules affect.

If you select By age, then you retain images based on their age. Or, you can use Retain at least one image per recipe to keep only the latest available image. If you select By Count, then you can use Retain a specific number of images per recipe to specify how many images to retain.

Check whether Image Builder keeps images based on the AGE and COUNT filters.

In the following example output of get-lifecycle-policy, Image Builder deletes images only after 3 years:

"filter": {
    "type": "AGE",
    "value": 3,
    "unit": "YEARS",
    "retainAtLeast": 1
}

In the following example output of get-lifecycle-policy, Image Builder keeps 4 images:

"filter": {
    "type": "COUNT",
    "value": 4
}

If needed, then update the By age or By count filters of your lifecycle policy.

You can also use exclusion rules to keep images based on tags. Check whether Image Builder keeps images based on your exclusion rules.

In the following example output of get-lifecycle-policy, Image Builder keeps images that have the example-tag tag:

"exclusionRules": {
    "tagMap": {
        "RetainImageTag": "example-tag"
    }
}

To delete images with the lifecycle policy, remove the tag that's listed in RetainImageTag from the AMIs, snapshots, or container images.

Check the lifecycle policy scope

Recipes and tags in the Scope tab must match existing resources for the lifecycle policy to affect them.

If the tags and recipes don't match any resources, then you receive the following output when you run the get-lifecycle-execution command:

{    "lifecycleExecution": {
        "lifecycleExecutionId": "lce-#",
        "resourcesImpactedSummary": {
            "hasImpactedResources": false 
        }

If hasImpactedResources is false, then the runtime Amazon Elastic Compute Cloud (Amazon EC2) instance didn't identify any image resources to remove. To resolve this issue, make sure that the lifecycle policy scope includes recipes and tags that affect the available image resources.

Check whether you correctly tagged your Image Builder images

Image Builder runs the lifecycle policy based on tags for only Image Builder image resources. You can't use AMI or recipe tags for the lifecycle policy's scope. Make sure that you correctly tagged the Image Builder resources with the tags that you specified in the lifecycle policy's Scope tab.

To tag an existing image, run the following tag-resource command:

aws imagebuilder tag-resource --resource-arn example-image-resource-arn --tags KeyName1=Value1,KeyName2=Value2 --region example-region

Note: Replace example-image-resource-arn with the image ARN, Value1 and Value2 with the tags, and example-region with your Region. You must use the AWS CLI to tag Image Builder image resources.

Check your cross-account configuration

To allow Image Builder to perform lifecycle actions in another AWS account, create an IAM role to perform lifecycle actions in the destination account.