RDS Blue/Green Deployment Caused External Connectivity Failure (DNS Resolution Issue)

Question

Severity: High (Production environment impact)

### 1. Problem Description

Since completing a Blue/Green deployment, we are unable to establish new connections to our Production RDS MySQL instance from the public internet (SQL tools, new application connections). Connections stopped working immediately after the switchover.

The root cause appears to be an incorrect IP address resolution for the RDS endpoint, specifically when trying to connect from outside the VPC.

### 2. Diagnosis Details & Evidence

- RDS Configuration: "Publicly Accessible" is set to YES. Security Group, NACL, and Route Tables are all confirmed to allow public access on port 3306.

- Test 1: Internal Connection (EC2 in the same VPC)
  - Result: SUCCESS (telnet resolves and connects to a Private IP: 10.0.x.x)
  - Conclusion: DB instance is alive and reachable internally.

- Test 2: External Connection (Local/Other VPC)
  - Result: FAILURE (telnet resolves to an external IP, e.g., 43.202.7.160, then hangs/timeouts.)
  - Result (Application): java.net.SocketTimeoutException: Connect timed out.
  - Conclusion: New external connections are blocked/unanswered.

- Contradictory Behavior: Other RDS instances (Dev) using the exact same Public/Private mixed Subnet Group connect fine from local. Also, existing application connections on the Production server remain alive.

### 3. Suspected Root Cause

We strongly suspect a failure in the **AWS RDS DNS System** following the Blue/Green switchover:
The RDS endpoint DNS record is either pointing to an old/defunct Public IP or is incorrectly resolving to a Private IP for external lookups, despite the instance being publicly accessible. This is an **AWS Internal DNS Mapping Error.**

Please help, anybody.

Vinuthna Korinni · Answer

Try clearing your DNS cache on affected systems. This often resolves immediate connection issues.
1. On Windows, use ipconfig /flushdns, 
2. Mac -> use sudo killall -HUP mDNSResponder, 
3. Linux -> use sudo systemd-resolve --flush-caches.

If DNS clearing doesn't help, consider these temporary solutions:
Create a new read replica and promote it to master
Use an alternative endpoint temporarily

To confirm this suspected root cause, we can verify:
1. Compare DNS resolution from inside and outside VPC
dig +trace your-rds-endpoint.region.rds.amazonaws.com
2. Verify Endpoint History - Capture the endpoint information over timeaws rds describe-db-instances --db-instance-identifier your-instance-id
3. Test connection using external DNS server (8.8.8.8)

Consider implementing an RDS Proxy or connection pooling if you frequently perform Blue/Green deployments to help manage connections more effectively during switchovers.

Yeon Kang · Answer

1.  **System Status:** RDS instance is "Available," has Publicly Accessible = "Yes," and all Security Group/NACL/Route Table settings are confirmed to be correct. Existing application connections inside the VPC are running fine.

2.  **NSLOOKUP Result (The Core Problem):**
    * Command: `nslookup [RDS-ENDPOINT]`
    * Result: Resolves to **43.202.7.~~~** (An old/defunct Public IP).

3.  **Connection Behavior Confirmation:**
    * **External Access (Fails):** Local PCs/Other VPCs attempt to connect to **43.202.7.~~~** and fail (Connect timed out).
    * **Internal Access (Succeeds):** An EC2 instance within the VPC bypasses the public DNS error and successfully connects to the actual active **Private IP: 10.0.6.~~~** (Verified via successful telnet).

RDS Blue/Green Deployment Caused External Connectivity Failure (DNS Resolution Issue)

1. Problem Description

2. Diagnosis Details & Evidence

3. Suspected Root Cause

Add your answer

Relevant content