How do I verify that EC2 Auto Scaling lifecycle actions complete in AWS CodeDeploy?

Short description

CodeDeploy uses Amazon EC2 Auto Scaling lifecycle hooks to perform deployments. Lifecycle hooks might influence the sequence of Amazon EC2 instance lifecycle states because:

• Failed deployments cause related lifecycle actions to fail.
• Maximum number of concurrent deployments associated with the account.
• Multiple deployment groups associated with a single Amazon EC2 Auto Scaling group that exceed CodeDeploy limits and cause timeouts.
• Auto Scaling group tried to launch too many Amazon EC2 instances with lifecycle hooks that caused mandatory APIs to be throttled.
• Leftover lifecycle hooks associated with earlier deployment groups.

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Review your Auto Scaling group scaling activities history

1. In the EC2 console navigation pane, under Auto Scaling, choose Auto Scaling Groups.
2. In the content pane, select your Auto Scaling group, and then choose the Activity History view.
3. In the AWS CLI, run the command describe-scaling-activities.
4. Check for the following issues in the history:

Failed deployments

The following error shows a failed deployment with CodeDeploy:

Description: Launching a new EC2 instance: i-f362a82e. Status Reason: Instance failed to complete user's Lifecycle Action: Lifecycle Action with token 0fc62a4b-153b-4dba-9bf4-9f233e72a697 was abandoned: Lifecycle Action Completed with ABANDON Result Cause: At 2016-07-22T16:16:56Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 2 to 3.

Check your deployment details to review the deployment failure:

1. In the CodeDeploy console, under Deploy, choose Deployments, then select the failed Deployment Id.
2. To check for additional information within the Lifecycle Events, select View events.
   Note: When a CodeDeploy deployment fails, an ABANDON signal is sent to the Auto Scaling group to stop the instance. For more information on troubleshooting a failed deployment, see Troubleshooting CodeDeploy and CodeDeploy quotas.

Note: Each deployment associated with a scaled-up Amazon EC2 instance counts as a singular concurrent deployment within the AWS Account.

Heartbeat timeout issues within the lifecycle hook

The following error shows that the hook timed out because of the five-minute limit set by CodeDeploy within the lifecycle hook configuration.

Description: Launching a new EC2 instance: i-649f40a2. Status Reason: Instance failed to complete user's Lifecycle Action: Lifecycle Action with token 95c60857-3c0a-4a67-8518-6dc18797e421 was abandoned: Heartbeat Timeout Cause: At 2016-07-23T05:08:35Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 1 to 2.

This timeout can occur because:

• Multiple deployments send commands to the CodeDeploy agent at the same time. To resolve this error: Consolidate your deployment instances in a single deployment group. Be sure to delete unnecessary deployment groups to prevent unnecessary lifecycle hooks. For more information, see Working with deployment groups in CodeDeploy.
• The following mandatory APIs are throttled: autoscaling:RecordLifecycleActionHeartbeat or autoscaling:CompleteLifecycleAction. To resolve this error: Check AWS CloudTrail, and then filter EventName=RecordLifecycleActionHeartbeat or EventName=CompleteLifecycleAction. If throttling events are logged, space out automatic scaling scale-outs in the account. Or, create a case with AWS Support to ask for a limit increase.

Leftover lifecycle hooks from deleted applications

Check for active lifecycle hooks associated with deployment groups that have deleted applications. This issue can occur when an application is deleted before its deployment groups are deleted. For more information see, EC2 instances in an Amazon EC2 Auto Scaling group fail to launch and receive the error "Heartbeat Timeout."

For more information about completed lifecycle hook results, check for --lifecycle-action-result parameter in complete-lifecycle-action API call in CloudTrail.

Related information

Integrating CodeDeploy with Amazon EC2 Auto Scaling