Understanding RDS Proxy Connection Persistence with Amazon EKS: Causes and Solutions

Issue Overview

When using Amazon RDS Proxy with Amazon EKS, you may notice that proxy connections remain active even after EKS pods are terminated. This article explains this behavior and provides guidance on managing these connections effectively.

RDS Proxy Connection Management

RDS Proxy manages two types of connections:

1. Client connections (Client ↔ Proxy)
2. Database connections (Proxy ↔ Database)

Client <-----> RDS Proxy <-----> Database

The proxy uses connection pooling to maintain and reuse database connections for improved performance. When a client (like an EKS pod) connects through the proxy endpoint, it uses an available connection from this pool.

Key Connection Management Parameters

1. IdleClientTimeout

• Controls how long a client connection can remain idle before closure
• Default: 1,800 seconds (30 minutes)
• Location: RDS console under "Idle client connection timeout"
• When to adjust:
  • Increase if your workload establishes frequent connections
  • Decrease to proactively remove stale connections

2. MaxIdleConnectionsPercent

• Controls the percentage of idle database connections in the connection pool
• Default: 50 percent
• Maximum: Limited by MaxConnectionsPercent
• When to adjust:
  • Higher values better handle sudden activity surges
  • Lower values reduce idle connection count

Connection Pruning Process

Connections are marked for pruning when:

• They exceed IdleClientTimeout
• They reach the 24-hour age limit
• Idle connections exceed MaxIdleConnectionsPercent

Pruning priority order:

1. Oldest connections first
2. Idle connections over active ones
3. Connections without active transactions

Best Practices for Connection Management with EKS

1. Gradual Configuration Changes

   • When reducing MaxIdleConnectionsPercent, implement changes gradually
   • Recommended progression: 50% → 40% → 30% → 25%
   • Monitor application performance after each change

2. Workload Considerations

   • For predictable EKS workloads: Lower MaxIdleConnectionsPercent is generally safe
   • For variable workloads: Maintain higher values to handle traffic spikes

3. Monitoring

   • Track connection counts and latency
   • Monitor for connection errors during peak loads
   • Adjust settings based on observed patterns

4. EKS-specific Considerations

   • Align IdleClientTimeout with your pod lifecycle expectations
   • Consider using connection pooling at the application level within your pods

Why Connections Persist After Pod Termination

RDS Proxy doesn't immediately detect client-side disconnections (like pod termination) because:

• TCP connections can terminate ungracefully
• RDS Proxy relies on timeout mechanisms rather than active connection checking
• This design helps handle temporary disconnections and rapid reconnections

Additional Resources

• RDS Proxy Connection Configuration
• Connection Pooling with RDS Proxy
• EKS Best Practices Guide

Remember to validate these settings in a test environment before applying them to production workloads.

Article Co-Auther:

• Rakesh Reddy Kondakindi