Heath check failing only after enabling VPC connector

Double check the default security group. This is usually very restrictive for inbound connectivity. Use your own SG as best practice is not to use the Default VPC and any default Security groups

Based on the information provided, it appears that the issue is related to the VPC connector configuration for one of your App Runner services. Here are some potential reasons and suggestions to troubleshoot the problem:

1. Security Group Configuration: Even though you're using the default VPC security group, it's possible that the security group rules are not allowing the necessary inbound traffic for the health check. Ensure that the security group associated with your VPC connector allows inbound traffic on port 3000 (the port your application is listening on).

2. Network ACLs: Check if there are any Network ACLs in your VPC that might be blocking the health check traffic to your application.

3. Application Behavior: Since one service works and the other doesn't when using the VPC connector, it's possible that the problematic service is behaving differently when it detects it's running in a VPC environment. Review your application code to ensure it's not altering its behavior based on the network environment.

4. Subnet Configuration: Verify that the subnets you've selected for your VPC connector have the necessary route table entries to allow traffic to and from your application.

5. DNS Resolution: Ensure that DNS resolution and DNS hostnames are enabled for your VPC, as App Runner may rely on DNS for health checks.

6. Application Startup Time: If your application takes longer to start up when running in the VPC environment, it might not be ready when the health check is performed. You could try increasing the health check interval or timeout in your App Runner service configuration.

7. VPC Endpoint: If your application depends on other AWS services, ensure you have the necessary VPC endpoints set up for those services.

8. Logs: Although you mentioned that no application logs are published to CloudWatch, double-check your CloudWatch settings to ensure logging is properly configured. Application logs could provide valuable insights into why the health check is failing.

To further troubleshoot, you could try:

• Temporarily changing the health check from TCP to HTTP if your application supports it, which might provide more detailed error information.
• Deploying a simple test application that just responds to health checks to isolate whether the issue is with your application or the VPC configuration.
• Comparing the detailed configurations of both services to identify any subtle differences that might be causing the issue.

If none of these suggestions resolve the issue, you may need to contact AWS support for more in-depth troubleshooting of your specific configuration.
Sources
Enabling Private endpoint for incoming traffic - AWS App Runner
Using App Runner with VPC endpoints - AWS App Runner
Terminology - AWS App Runner

Hey thanks for your answer. The security group is currently the same for ingress and egress - 0.0.0.0/0 and ::/0 - all protocols/ports. I also tried changing the health check to HTTP and had the same results (one fails, one passes). DNS resolution and hostnames are enabled for the VPC. Honestly I struggle seeing it being a VPC issue, since there is one App Runner service that works completely fine using the same VPC connector. This fact does eliminate a lot of possibilities.

Ok so after posting this I think I managed to solve it (somewhat) - but I think there may be some room for improvement with the error messages provided by AWS.

I created a brand new VPC to try creating the App Runner app with a connection to a completely new VPC instead of the default one. I'm using Terraform for pretty much everything, and so previously, I created the VPC connector using Terraform for my default VPC. This time however, while I created the VPC using Terraform, I used the dashboard to create the App Runner app along with the VPC connector. I created this new VPC with public subnets only, and when creating the app, AppRunner would not let me choose the public subnets, only private subnets (of which there were none).

This immediately seemed strange to me, because the VPC connector I created via Terraform for the default VPC was using public subnets (my default VPC has only public subnets). I went to check again - yes, my original VPC connector is definitely using public subnets. I've now searched the documentation for information relating to subnets and I found this:

When selecting a subnet for your VPC, ensure that you choose a private subnet, not a public subnet.

It appears that the AWS Dashboard prevents you from accidentally selecting a public subnet, but the AWS API does not.

I created private subnets and a NAT gateway for my new VPC using Terraform, and when I went back to App Runner it allowed me to create the VPC connector with the private subnets. Now, when I create the App Runner service, it starts successfully.

So it seems that the answer here is that the VPC connector needs to be set up in private subnets only. Is there a use case for using public subnets? If not, I think the AWS API should not allow selecting public subnets. Another thing I don't quite understand is why did one service consistently work and the other one consistently didn't? This is what really threw me off the mark debugging this issue.