How do I resolve the error "Failed to receive X resource signal(s) within the specified duration" in AWS CloudFormation?

Short description

You get this error when an Amazon Elastic Compute Cloud (Amazon EC2) instance, Auto Scaling group, or a WaitCondition doesn't receive success signals from one or more instances in the time period specified by the CreationPolicy attribute.

This error can occur in one of the following scenarios:

• Scenario 1: The cfn-signal script isn't installed on one or more instances of the AWS CloudFormation stack.
• Scenario 2: There are syntax errors or incorrect values in the AWS CloudFormation template.
• Scenario 3: The value of the Timeout property for the CreationPolicy attribute is too low.
• Scenario 4: The cfn-signal isn't sent from the Amazon EC2 instance.

Note: The troubleshooting scenarios for this error apply only to AWS CloudFormation stacks created with Linux instances. The scenarios don't apply to Windows instances. For more information, see How to troubleshoot stack creating issues.

Resolution

Before following the steps in the troubleshooting scenarios, set the Rollback on failure option for your AWS CloudFormation stack to No.

Scenario 1: The cfn-signal script isn't installed on one or more instances of the AWS CloudFormation stack

To confirm that the cfn-signal script is installed on the instance that is configured to send signals to AWS CloudFormation resources, complete the following steps:

1.    Connect to your Linux instance using SSH.

2.    Confirm that the cfn-signal script is installed using either of the following commands.

To confirm that the cfn-signal script is located in your directory, run the following command:

$ sudo find / -name cfn-signal
/opt/aws/bin/cfn-signal
/opt/aws/apitools/cfn-init-1.4-30.amzn2/bin/cfn-signal

To confirm that the AWS CloudFormation helper scripts package that contains the cfn-signal script is installed, run the following command:

$ sudo rpm -q aws-cfn-bootstrap
aws-cfn-bootstrap-1.4-30.amzn2.noarch

Important: The preceding command works only on distributions that use the RPM Package Manager.

Note: By default, AWS CloudFormation helper scripts are installed on Amazon Linux Amazon Machine Images (AMIs). If AWS CloudFormation helper scripts aren't installed, see CloudFormation helper scripts reference for installation instructions.

Scenario 2: There are syntax errors or incorrect values in the AWS CloudFormation template

To confirm that the UserData property is configured to signal the AWS CloudFormation resources specified by the CreationPolicy attribute, complete the following steps:

1.    In a code editor, open the AWS CloudFormation template for your stack, and then find the UserData property section.

2.    Check for errors, including syntax errors, missing spaces, misspellings, and other typos.

3.    Confirm that the values for the stack, resource, and Region properties are correct.

Note: If you use a bootstrap script that includes the UserData property and calls the cfn-signal script, then check the bootstrap script for syntax errors or incorrect values.

If you signal within the cfn-init commands key, look for information about the signal in the cfn-init logs. To search for errors in the cloud-init logs or cfn-init logs, connect to your Amazon EC2 instance using SSH. Then, look for detailed error or failure messages by searching for the keyword "error" or "failure" in the following logs:

/var/log/cloud-init-output.log
/var/log/cloud-init.log
/var/log/cfn-init.log
/var/log/cfn-init-cmd.log
/var/log/cfn-wire.log

To parse all instances of the words "error" or "failure" in any /var/log/cfn or /var/log/cloud-init files, run the following command:

grep -ni 'error\|failure' $(sudo find /var/log -name cfn\* -or -name cloud-init\*)

Note: The preceding command returns the file name, line number, and error message.

Scenario 3: The value of the Timeout property for the CreationPolicy attribute is too low

The value of the Timeout property is defined by the CreationPolicy attribute. To confirm that the value is high enough for tasks to run before the cfn-signal script sends signals to AWS CloudFormation resources, complete the following steps.

Important: The following steps work only if the instance isn't terminated (for example, by an Auto Scaling group). You already set the Rollback on failure option of your AWS CloudFormation stack to No. This option means that there is no failure rollback, and the instance won't be terminated until you delete the stack. You can connect to the instance using SSH, and then continue with the following troubleshooting steps.

1.    In a code editor, open the AWS CloudFormation template for your stack, and then find the value of the Timeout property.

Note: The value of the Timeout property is the maximum amount of time AWS CloudFormation waits for a signal before returning an error.

2.    To get an estimate of when the cfn-signal script is triggered, connect to the instance using SSH, and then run the following command:

less /var/log/cfn-init.log

The log file shows a timestamp when the SUCCESS signal is sent to AWS CloudFormation resources. See the following example:

2019-01-11 12:46:40,101 [DEBUG] Signaling resource EC2Instance in stack XXXX with unique ID i-045a536a3dfc8ccad and status SUCCESS

3.    Open the AWS CloudFormation console.

4.    To see the resource failure timestamp for the "Failed to receive X resource signal(s) within the specified duration" event, choose the Events view.

5.    For Status Reason, expand the row for the event with the status reason "Failed to receive X resource signal(s) within the specified duration."

6.    Compare the signaling timestamp to the resource failure timestamp.

Note: Notice that the signal was sent after the Amazon EC2 resource failed to be created. The signal is sent before the Amazon EC2 resource is created or fails to be created.

Scenario 4: The cfn-signal isn't sent from the Amazon EC2 instance

The SignalResource API is useful when you want to send signals from anywhere other than from an Amazon EC2 instance.

For example, you can use an AWS Lambda function to call the SignalResource API and then send the signal to the AWS CloudFormation stack. In such a scenario, check your Lambda logs with Amazon CloudWatch Logs. These logs help you figure out why the signal isn't sent to the AWS CloudFormation stack.