App Runner unable to access internet or RDS on initial startup

Question

Hi I have been trying to implement a Spring boot app in App runner and deployed as part of 3 part Cloudformation stack using AWS CDK v2. Consisting of VPC-Stack, RDS-Stack and AppRunner-Stack. It is a fairly standard VPC design. IGW, 2 Public, 2 Private and 2 isolated Subnets. Nat gateway in Public Subnet, AppRunner in private subnet(s), RDS in isolated subnet(s). App runner VPC connector is used to connect from App runner to RDS and set security groups etc.

The problem is the AppRunner-stack which includes a VPC connector is very flaky on startup and cannot initially connect to internet or RDS. It takes hours until the vpc connect is able to connect to internet and rds. I have engaged AWS support and their initial response was to inform of this but did not offer any reason why, work arounds, plan or roadmap to fix it (I've prompted and i'm awaiting further clarification)

To debug what is going on I basically now have a simple testing container built in python which tests internet connectivity, database connectivity and serves a demo flask app.

To i "cdk deploy" the initial stacks including app runner. then repeatedly re-deploy the test container awaiting for internet and rds connectivity to work. It takes hours! once established subsequent manual deploys work fine. I've tested this in ap-southeast-2, us-east-1 and us-west-2.

Is anyone aware what could be going on, how I could work around this issue (preferably with IaC/pipelines and automation) rather than being forced to hand hold the damn thing until the connections stabalise!, also, any ideas on whether AWS have a plan to fix this? I have not found anything online about it or in github issues roadmap

Answer

Thank you for the details. I should have mentioned I am using VPC Connector within a private subnet. It has security group associated to specifically grant access to the RDS. I also have a Bastion host within the same private subnet and it does immediately have connectivity to the database and internet. Again, the issue is that the connectivity is sporadic and once connectivity starts working say after 1 hour, I can repeatedly tear down the AppRunner stack (including VPC Connector) and connectivity will work flawlessly. Something very strange is going on...

I've checked VPC Flow logs and see no DENYs, I have not however checked the ENI flow logs. In fact, I've just had a check of this, that is great advice yes, I see my AppRunner ENI's I'll create flow logs for those and monitor for DENYs etc

Of course this is just an issue on initial deployment of infra but of course I want pushbutton pipeline deployment I don't want to need to manually intervene, especially when (the unlikely) issue occurs that an incident occurs and due to DR and recovery I need to urgently redeploy a stack in another region etc

Let me check into ENI logs...

Answer

The VPC connectivity issues you described with AppRunner are likely due to networking configuration problems. A few things you can try:

Ensure the subnets used in the VPC Connectors are private subnets, not public. Public subnets will cause errors.

Verify security group ingress/egress rules on the VPC Connector and downstream resources like RDS allow communication between them.

Test connectivity directly from an EC2 instance in the same VPC to rule out issues outside of AppRunner.

If direct tests fail, contact AWS Support to troubleshoot further.

If direct tests succeed, enable ENI flow logs to check if traffic is entering the ENI:

- Launch the EC2 console and go to Network Interfaces 
- Select the AppRunner ENIs 
- Choose "Create flow log"

This will help identify if the issue is at the ENI level. No official workaround currently but enabling flow logs is one approach to gather more insights. The underlying cause could be transient networking or configuration issues. I'd check the AppRunner documentation or contact Support for the latest recommendations.

App Runner unable to access internet or RDS on initial startup

Relevant content