High number of connections in DocumentDB

Based on your description, you're experiencing an issue with a high number of connections to your DocumentDB cluster that doesn't seem to correlate with actual Lambda invocation activity. This is a challenging situation that requires investigation on multiple fronts.

When using PyMongo with Lambda functions, each Lambda container can maintain its own connection pool to DocumentDB. By default, PyMongo creates a connection pool with a maxPoolSize setting that determines the maximum number of connections maintained in the pool. If you have many Lambda functions or high concurrency, these connection pools can accumulate and potentially exceed your instance's connection limits.

The "active": false connections you're seeing in db.currentOp(true) are likely idle connections being maintained in connection pools. Even if the Lambda functions are dormant in terms of invocations, the containers might still be warm and maintaining open connections.

To identify which Lambda functions are connecting to your DocumentDB:

1. Check CloudTrail logs for API calls to your DocumentDB cluster from Lambda functions.

2. Implement connection pooling with appropriate maxPoolSize settings in your Lambda functions to limit the number of connections each function maintains.

3. Consider implementing a connection monitoring solution where each Lambda function logs its connection activity to help track which functions are creating connections.

4. Review your Lambda code to ensure connections are being properly reused rather than creating new ones for each invocation.

For a more scalable solution, you might want to consider implementing a connection pooler like mongobetween. This acts as a middleware that can handle a large number of incoming connections from your Lambda functions and multiplex them across a smaller connection pool to your DocumentDB cluster. This approach would allow your application to scale independently of DocumentDB connection limits.

The saw-tooth pattern you're observing in read operations that follows the connection pattern suggests that connections are being established for read operations and then potentially not being properly closed or reused.
Sources
Scale your connections with Amazon DocumentDB using mongobetween | AWS Database Blog
Troubleshooting connection issues - Amazon DocumentDB

This is a complex issue that appears to be related to connection pooling and potentially leaked connections. Let me help break this down and suggest ways to investigate:

1. Connection Pooling Behavior:

• PyMongo 4.0.1 uses connection pooling by default
• The default maxPoolSize is 100 connections per MongoClient instance
• If connections aren't being properly closed, they can accumulate even with low activity

2. Finding the Source Lambda Functions: Here are several approaches to identify the connecting Lambdas:

a) VPC Flow Logs:

# Enable VPC Flow Logs if not already enabled
aws ec2 create-flow-logs --resource-type VPC --resource-id vpc-xyz \
    --traffic-type ALL --log-destination-type cloud-watch-logs \
    --log-destination arn:aws:logs:region:account:log-group:/vpc/flow-logs

Then analyze the logs to match the ENI IPs you found with actual Lambda traffic.

b) X-Ray Tracing:

• Enable X-Ray tracing on your Lambda functions
• Add the MongoDB X-Ray tracker to your Python code:

from aws_xray_sdk.core import patch_all
patch_all()

c) Add connection logging to your Lambda functions:

import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

def lambda_handler(event, context):
    logger.info(f"Function {context.function_name} connecting to MongoDB")
    # Your connection code here

3. Check Lambda Connection Configuration: Make sure your Lambda functions are properly closing connections:

from pymongo import MongoClient

def lambda_handler(event, context):
    client = MongoClient(your_connection_string)
    try:
        # Your database operations
        pass
    finally:
        client.close()

4. Monitor Active Connections: Run this command in MongoDB shell to monitor active connections:

db.currentOp(
  {
    "active" : false,
    "waitingForLatch" : false,
    "client" : {$exists: true}
  }
).forEach(function(op) {
  print(op.client + " - " + op.connectionId)
})

5. Immediate Mitigation Steps:

• Set explicit maxPoolSize in your Lambda functions:

client = MongoClient(your_connection_string, maxPoolSize=10)

• Add connection timeouts:

client = MongoClient(your_connection_string, 
                    serverSelectionTimeoutMS=5000,
                    connectTimeoutMS=5000)

6. Long-term Solutions:

• Implement connection pooling at the application level using a service like AWS RDS Proxy
• Use a connection management wrapper:

class MongoDBConnection:
    def __init__(self, connection_string):
        self.client = MongoClient(connection_string, maxPoolSize=10)
        
    def __enter__(self):
        return self.client
        
    def __exit__(self, exc_type, exc_val, exc_tb):
        self.client.close()

# Usage
with MongoDBConnection(connection_string) as client:
    db = client.your_database
    # Your operations here

7. Additional Investigation Tools:

• CloudWatch Logs Insights to correlate Lambda executions with connection spikes
• AWS CloudTrail to check for any configuration changes that might have triggered this
• DocumentDB Performance Insights (if enabled) to analyze connection patterns

Given that you're seeing low CPU utilization and inconsistent correlation with Lambda invocations, this strongly suggests a connection pooling or connection leakage issue rather than actual workload increase. The saw-tooth pattern you're seeing is typical of connection accumulation followed by timeout-based cleanup.

I'd recommend starting with implementing proper connection closing in your Lambda functions and setting explicit maxPoolSize values, then monitoring the situation for 24-48 hours to see if the connection count stabilizes.

Thank you for the suggestions. I have tried many of those with no success. The issue finally went away with no changes to our system and its stable as before the issue. Could this be something within the AWS DocumentDB service? Following is a chart of the database connections before, during the issue and afterwards.