How do I resolve errors for the failed CloudWatch canary in my VPC?

Short description

When you create a canary in a VPC, Amazon CloudWatch Synthetics automatically creates an AWS Lambda function with an elastic network interface in the specified VPC subnets. These elastic network interfaces don't receive public IP addresses by default even if you add them to public subnets.

To write logs to Amazon Simple Storage Service (Amazon S3) and publish data points to CloudWatch, the canary must have access to the Amazon S3 endpoint and CloudWatch Network Monitoring endpoint.

If your canary can't access either of these endpoints, then you receive one of the following error messages:

"TimeoutError: Navigation Timeout Exceeded:30000ms exceeded"

"No test result returned. Connection timed out after 30000ms and No artifacts were uploaded"

To test your canary's internet connectivity, use VPC Reachability Analyzer.

Resolution

Resolve "Navigation timeout" errors

You must correctly configure your security group, network access control list (network ACL), and routing table to allow access to the canary's endpoint. The configured timeout period for your monitored endpoint must be greater than the endpoint response time.

Confirm that the security group that's attached to the canary contains the following properties:

• The security group must have an outbound rule that allows connections between the configured port and the IP address of the monitored endpoint.
• The security group must allow outbound traffic to port 443. Amazon S3 and CloudWatch Monitoring endpoints are reachable through HTTPS. If an outbound rule doesn't exist, then add an outbound rule. When you add an outbound rule, use HTTPS (TCP port 443) for Type and 0.0.0.0/0 for Destination.
• The network ACL in the canary subnet must allow inbound and outbound access.
  For inbound access, allow ephemeral ports to access the source address and port of your endpoint IP address. Allow ephemeral ports to access 0.0.0.0/0 for port 443.
  For outbound access, allow access to the Destination address and port of your endpoint IP address. Allow the Destination access to 0.0.0.0/0 for port 443.
• Open the canary, and then choose Additional configurations to confirm that your endpoint responds within the configured timeout period.

Then, run the following command to find your endpoint response time:

time curl http/(s)://YOUR ENDPOINT IP/DNS:PORT

Note: Replace YOUR ENDPOINT IP/DNS with your DNS or IP address and PORT with your TCP port number.

Lastly, in the canary code, change the heartbeat timeout value to exceed your endpoint response time:

const response = await page.goto(URL, {waitUntil: 'domcontentloaded', timeout: 30000});

For example, if your configuration is "const response = await page.goto(URL, {waitUntil: 'domcontentloaded', timeout: 30000});" and your endpoint takes 35 seconds to respond, then change the timeout value to exceed 35000 milliseconds.

Resolve "No artifacts were uploaded" or "No test result returned" errors

If there are no data points in canary metrics, then your canary requires access to the CloudWatch Network Monitoring endpoint. If there are data points in canary metrics, then your canary requires access to the Amazon S3 endpoint. Note that 30000ms reflects the timeout period that you configured on the canary.

To allow your canary to access Amazon S3 and CloudWatch Monitoring endpoints, choose one of the following approaches:

• Use a NAT gateway to add the canary to private subnets and route traffic from the private subnet to the internet.
• Use VPC endpoints add the canary to private subnets to allow private connectivity to AWS services.

Use a NAT gateway

Create a NAT gateway. Then, update the canary's private subnet routing table. For information on NAT gateways as a destination in a route table, see Routing to a NAT device.

Use VPC endpoints

Create a gateway endpoint for the Amazon S3 endpoint.
Note: For Service name, search for Amazon S3, and then select com.amazonaws.region.s3.

Then, create an interface VPC endpoint for the CloudWatch Network Monitoring endpoint. For Additional settings, select Enable DNS name.
Note: For Service name, search for monitoring, and then select com.amazonaws.[region].monitoring.