How can I resolve the "Volume cannot be modified in modification state OPTIMIZING" error in CloudFormation when I try to update my Amazon EBS volume type (AWS::EC2::Volume)?

Short description

You get this error in AWS CloudFormation when you try to modify an Amazon Elastic Block Store (Amazon EBS) volume during a stack update. The error occurs because you use an AWS Identity and Access Management (IAM) user or role, and the volume update times out after seven hours. Instead, use an IAM service role that increases the volume timeout to 36 hours.

Note: The volume's timeout period applies only to volume modification that uses CloudFormation. The timeout period varies for different types of resources and actions.

Typically, a fully used 1-tebibyte (TiB) EBS volume takes about six hours to migrate to a new performance configuration. For more information see, Monitor the progress of volume modifications.

Important: If you already initiated a stack update with CloudFormation, wait for about seven hours until the stack stabilizes. Your stack is stabilized when it enters UPDATE_ROLLBACK_FAILED status. Then, complete the following steps.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Recover your stack and sync it with the volume

Complete the steps in the following sections:

Recover the stack

If you change your EBS volume type from st1 to gp3, then your stack can fail to update the volume type because of volume size.

Wait until your stack gets into an UPDATE_ROLLBACK_FAILED status. Then, use AWS CLI or the CloudFormation console to recover your stack. In this scenario, use the continue-update-rollback operation on your stack to perform an update.

Note: For the recovery operation, skip the logical ID of the resource that failed to update.

AWS CLI:

aws cloudformation continue-update-rollback --stack-name your-stack-name --resources-to-skip logical-ID-of-EBS-volume

CloudFormation console:

1. Open the CloudFormation console.
2. From the navigation pane, choose Stacks.
3. From the Stack name column, select the stack that's stuck in UPDATE_ROLLBACK_FAILED status.
4. Choose Stack actions, and then choose Continue update rollback.
5. In the Continue update rollback dialog box, expand Advanced troubleshooting.
6. In the Resources to skip - optional section, select the resources that you want to skip.
   Note: To skip the volume, select the logical ID of the volume that failed to update.
7. Choose Continue update rollback.

Update your stack to match the current status of your EBS volume

You must update your stack to match the current status of your EBS volume. Your CloudFormation template is updated by your stack update, and your stack changes to UPDATE_COMPLETE status.

For example, you can modify the volume type (AWS::EC2::Volume) from st1 to gp3 during a stack update operation. However, the volume type in CloudFormation template remains st1 because of the stack rollback operation, even if EBS accurately reflects the update to gp3.

To initiate an update, complete the following steps:

1. To monitor the status of your EBS volume, use the Amazon Compute Cloud (Amazon EC2) console. For more information, see Monitor the progress of EBS volume modifications.

   Or, run the following command:

   aws ec2 describe-volumes-modifications --volume-ids your-volume-ID

   The volume type changes to gp3 in Amazon EBS and the volume is now available.

2. Update the CloudFormation stack with the volume type (AWS::EC2::Volume) as gp3. Then, perform the stack update operation.

3. Use drift detection to verify that your EBS volume resource (AWS::EC2::Volume) is in sync with your stack.

Update the EBS volume outside of CloudFormation

Your stack update can take longer than 36 hours if you don't use a service role or if the volume size is high (~8 TiB). In such cases, your stack update might fail even after you use a service role with 36 hours of stabilization. Then, you must wait until your stack stabilizes. To avoid the wait period, complete the following steps:

1. Use a Retain deletion policy to detach the EBS volume from your stack without deleting the volume.
2. Use the Amazon EC2 console to update your EBS volume outside of CloudFormation. For more information, see Modify an EBS volume using Elastic Volumes.
3. Import your EBS volume back into the stack.

Note: If you want to keep the volume in the stack at all times, then you must keep the volume size below a threshold. For example, if 8 TiB of data takes 64 hours, then you can estimate the maximum size for 36 hours and lower the data size to 4 TiB. This threshold adjustment allows you to complete your update within the allotted time.