Configured VPC NAT instances stopped working yesterday (03.03.2022, eu-central-1)

Are you able to reach the NAT instance from the EC2 instance in the private subnet?

As I redeployed the whole infrastructure I'm stuck in the CodePipeline building stage right now where I only test with CodeBuild's temporary instance if it is working. In CodeBuild's "Edit Environment" menu I can "Validate VPC settings" and it fails with the standard error which I had some time ago when setting up the VPC for the first time:

"The VPC with ID vpc-xxxx might not have an internet connection because CodeBuild cannot find the 0.0.0.0/0 destination for the target NAT gateway with subnet ID subnet-xxxx. Verify your VPC has an internet connection through the device with ID i-xxxx."

I could try and set up an instance in one of the private subnets but then I also need to set up SSH access and probably ICMP for it (right now I only have TCP (HTTP and HTTPS) enabled as inbound traffic to the NAT instances from within the VPC (as recommended in the AWS guide)). Do you think it is helpful to set up a test instance to maybe gain additional info?

All in all it is a bit suspicious that "CodeBuild cannot find the 0.0.0.0/0 destination for the target NAT gateway" even though the routes should be configured correctly and have worked until yesterday.

Yes i think it would be helpful to quickly spin up an EC2 instance in the same private subnet where codepipeline's temporary instance is. Then you can test connectivity to your NAT instance over ICMP then to the internet.

Since your test instance will be in the private subnet, you will need to also setup a Bastion in the public subnet to get to it. See https://aws-quickstart.github.io/quickstart-linux-bastion/#_launch_the_quick_start

OR setup SSM Session Manager which doesnt use SSH. See https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started.html

Another option is to use the Reachability Analyzer https://docs.aws.amazon.com/vpc/latest/reachability/what-is-reachability-analyzer.html

Hi mthwbarb, thanks for your quick answer!

1. I disable source/destination check in the template, so it is always turned off for the instances.

2. I have 3 custom route tables, one for each private subnet. They have the default entry plus a custom route: Destination Target 0.0.0.0/0 <NAT instance ID> (which gets resolved to the corresponding network interface of the instance: eni-xxxxxxxxxxxxxxxxx)

3. I understand that NAT gateways might be the easier and better solution but they are too expensive for my use case.

About the Amazon Linux 2 configuration, it is configured with user data in the template, e.g.: #!/bin/bash sudo bash -c 'echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf' sudo sysctl -p sudo yum -y update sudo yum install -y iptables-services sudo systemctl enable iptables sudo systemctl start iptables sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE sudo service iptables save sudo yum install -y aws-cfn-bootstrap sudo /opt/aws/bin/cfn-signal -e 0 --region ${AWS::Region} --stack ${AWS::StackName} --resource NatInstance1

The configuration is slightly different than described in the AWS guide to make the settings persistent between potential reboots. I changed "sudo sysctl -w net.ipv4.ip_forward=1" as shown above, some days before I was still using "sudo sysctl -w net.ipv4.ip_forward=1" and all other commands, only "sudo bash -c 'echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf'" and "sudo sysctl -p" was exchanged. Everything worked fine, now it is not working even when trying the exact configuration from the guide.