Why do my CloudFormation stack operations fail after I tag resources when I didn't change any IAM policies?

Resolution

In new AWS accounts, if you create, update, or delete resource tags in CloudFormation stacks without the correct tagging permissions, then your stack operations fail.

Existing accounts are exempt from the new stack operations behavior until February 29, 2024.

To prevent service disruptions, grant the necessary tagging permissions to your IAM principal. If your IAM principal can't manage resource tags, then CloudFormation performs stack operations and doesn't change any resource tags. As a result, stack tags might not match resource tags. To resolve this issue, you must synchronize your resource tags within your CloudFormation template.

Important: In the following resolution, some steps might be optional for your use case.

Add missing permissions to the IAM principal

Follow these steps to manage your resource tags with CloudFormation in existing or new accounts.

Find the IAM principal that's used for tagging operations

To find the IAM principal that you want to use to tag resources, complete the following steps:

1. Open the CloudFormation console.
2. In the navigation pane, choose Stacks.
3. Select your stack name, and then copy the stack ID.
4. Open the AWS CloudTrail console.
5. In the navigation pane, choose Event history.
6. For Lookup attributes, choose Resource name. Then enter the stack ID into the search box, and run the search.
   Note: Event history includes events from the past 90 days. If your stack ID search doesn't display any events, then update the stack. Stack updates are recorded in the CloudTrail event history. You can retrieve an IAM principal when an update is recorded.
7. Select the first event name of the Event history list, and then review the event record.
8. Find the IAM principal in the event payload.
9. Search for "requestParameters" to review the roleArn property. If there's an roleArn, then the string value is the IAM principal.
10. If there isn't a roleArn, then search for "userIdentity" to review the principalId value.

Grant tagging permissions in an IAM policy

When an infrastructure as code (IaC) service manages an IAM principal, you must update the related IAM policy with the same IaC service. To use the AWS Management Console to grant tagging permissions, create a new policy or update an existing policy. You can also run put-role-policy in the AWS Command Line Interface (AWS CLI) to configure the IAM principal's tagging permissions.

For more information, see How do I resolve the tagging permission error in my CloudFormation stack?

Important: To deploy CloudFormation stacks in a new account, skip the following section.

(Optional) Synchronize resource tags across your CloudFormation stacks in existing accounts

When you configure tagging permissions for your CloudFormation stack, your resource tags might not match the tags in your CloudFormation template. To match your template tags with your resource tags, complete the following steps.

Check the affected stacks for drift

To check your affected stacks for drift, complete the following steps:

1. Open your AWS Health Dashboard to get your affected resources data.
2. Detect drift on your CloudFormation stack and review the drift results.
   Note: For more information about drift detection, see Resources that support import and drift detection operations and Detect drift on individual stack resources.
3. A resource is correctly configured when your resource's drift status is IN_SYNC and the resource supports drift detection.
4. When a resource's drift status is MODIFIED, select the resource and choose View drift details to review the differences.
5. Depending on the drift scenario you encounter, complete the steps in the following sections.

Synchronize resource tags that support drift detection with your CloudFormation template

Synchronize resource tags that support drift detection in the following scenarios:

• A resource's drift status is MODIFIED and the resource supports drift detection.
• All resources have tag changes with the REMOVE difference-type status code.
• A resource has a tag change that's not labeled REMOVE, and each tag key in Actual is within Expected.

Important: These steps temporarily remove stack-level tags from other resources. If any of your services rely on stack-level tags, then see Fix the tags on the resource itself to avoid disruptions.

To synchronize your resources tags with your CloudFormation template, complete the following steps:

1. Save a copy of your original CloudFormation template.
2. In the copied CloudFormation template, remove all tags that are labeled REMOVE.
3. Update the stack with the new template, and then remove any stack-level tags that are labeled REMOVE.
   Warning: Tags are removed from all resources in the stack. Make sure to reapply the tags that failed.
4. Revert to the original template, and then redeploy the template with all the stack-level tags added back.
   Note: These updates reapply all of the tags that failed to apply when you didn't have sufficient permissions.

Fix the tags on the resource

Fix the tags on a resource in the following scenarios:

• A resource has a tag change that's labeled REMOVE. The resource's tag change indicates that the tag keys in the Expected state are missing in the Actual state.
• A resource has a tag change that's not labeled REMOVE. The tag keys are in both the Expected and Actual states, but with different values.

Review the following drift details on the resource:

• Note each tag key/value that's listed under Expected and missing from Actual. You must add these tags.
• Note each tag key that's listed under Actual, but missing under Expected. You must remove these tags.
• Note each tag key that's listed under Expected and Actual and has a different value. You must update these tags to Expected.

To fix the tags on the resource, complete the following steps:

1. Open the CloudFormation console.
2. In the navigation pane, choose Stacks.
3. Select the affected stack.
4. Choose Resources, and then choose the Physical ID link of the affected resource.
5. In the resource's Tags section, manually add, remove, or update each tag. The tags must match the state that's listed under Expected in the drift details.
6. Choose Save.

To verify that the stack and resources are in sync, it's a best practice to perform the drift detection operations again. To verify, complete the steps in the Check the affected stacks for drift section.

Synchronize the resource tags that don't support drift detection

Synchronize resource tags that don't support drift detection when a resource's drift status is IN_SYNC and the resource doesn't support drift detection.

Note: The following update reapplies all of the tags that failed to apply when you didn't have sufficient permissions.

If your resources don't support drift detection, then use a CloudFormation template to address inconsistencies and verify that CloudFormation manages the resources.

Complete the following steps:

1. Save a copy of your original CloudFormation template.
2. In the copied CloudFormation template, compare the resource tags to the deployed resources.
3. Depending on the tags you encounter, complete the steps that apply to your situation.

If no tags are specified in your resource and there are tags defined in the CloudFormation template, then complete the following steps:

1. Remove all tags from the CloudFormation template.
2. Apply the new CloudFormation template to update the stack. Then remove any stack-level tags.
3. Revert to the original CloudFormation template, and then redeploy the template with all the stack-level tags added back.

If the tags specified in the deployed resource don't match the tags in the stack, then complete the following steps:

1. Remove all of the stack-level tags to update the stack.
2. In the CloudFormation template, modify all the tags so that they match the deployed resource.
3. Update the stack with the new CloudFormation template. Make sure that all the previously associated stack-level tags are included.