How do I troubleshoot the "Canary Navigation Timeout Exceeded" error in CloudWatch?

Short description

The "Canary Navigation Timeout Exceeded" error occurs when a browser or endpoint can't connect to the canary within the specified time.

Note: When you create a CloudWatch canary, CloudWatch automatically creates an AWS Lambda function as part of the canary configuration. You can configure the canary timeout in the Additional configuration section on the CloudWatch console. Or, you can configure the Lambda function timeout separately in the Lambda console. If you set the Lambda function timeout lower than the canary script timeout, then the Lambda function times out before the canary script completes.

Resolution

Check endpoint reachability

Run the following command to verify that your canary reaches the specified endpoint and receives a response within the timeout period:

time curl https://example-endpoint-ip:example-port

Note: Replace example-endpoint-ip with your endpoint host name or IP address and example-port with your port.

If the monitoring endpoint uses a firewall or proxy, then verify that rules don't block the endpoint connection for the canary.

If you use an Amazon Virtual Private Cloud (Amazon VPC) canary, then check whether the canary reaches the endpoint. If the endpoint isn't reachable, then verify that security groups and network access control lists (network ACLs) allow access.

Verify canary script timeout values

Verify that the canary script timeout is greater than the endpoint response time. If the canary script timeout is less than the endpoint response time, then increase the timeout value in the canary script.

To change the canary heartbeat timeout from the canary code, run the following command:

const response = await page.goto(URL, {waitUntil: 'domcontentloaded', timeout: 30000}); 

Resolve navigation timeout issues in the GUI workflow builder

If you use the GUI workflow builder, then verify that you correctly configured the Click with navigation action. Then, confirm that the Click with navigation action on the button or link redirects to the correct destination. If your web page uses asynchronous operations, then increase the timeout value to accommodate the asynchronous operations.

Add the following code to your canary source code script to change the timeout for the Click with navigation action:

await Promise.all([             page.waitForNavigation({ timeout: 30000 }),  
           await page.click("example-button")

If the Click with navigation action doesn't redirect to the correct destination, then update your canary source code script to use the Click action instead.

After you update your canary source code script, run the following command to confirm that the endpoint responds within the configured timeout:

time curl https://example-endpoint-ip:example-port

Note: Replace example-endpoint-ip and example-port with your IP address and port number.

Verify VPC network configuration

If your canary runs in a virtual private cloud (VPC), then take the following actions:

• Verify that the security group for your canary has an outbound rule that allows traffic to the endpoint.
• Verify that the network access control list (network ACL) has inbound and outbound rules that allow traffic to the endpoint.
• Verify that the route table for the configured subnet has a route to the endpoint.
• Verify that a NAT gateway or elastic IP address is attached to the Lambda function for your canary.

Adjust timeout for complex or dynamic pages

If your canary navigates a dynamic page with extensive JavaScript or DOM elements, then the page requires more time to execute. Review your canary source code script for complex JavaScript or DOM elements that might cause the canary to time out before the page finishes loading. 

If your script contains extensive JavaScript or DOM elements, then increase the timeout value in your canary source code script. This gives the page enough time to load and execute.

Review CloudWatch metrics and logs

Review the CloudWatch metrics for your canary to identify performance patterns that correlate with timeout errors. Also review the CloudWatch logs that the Lambda function generates to find entries related to timeout errors.

Check the SuccessPercent, Duration, and Failed metrics for your canary. Also check for 4xx and 5xx HTTP error codes that signals endpoint or server issues.

To review the logs for your canary, complete the following steps:

1. Open the CloudWatch console.
2. In the navigation pane, choose Log groups.
3. Find the log group for your canary.
   Note: The log group name uses the /aws/lambda/cwsyn-[canary-name]-[uuid] format.
4. Review the log entries for timeout error messages, network connection failures, and script execution errors.

Related information

Troubleshooting a failed canary