MongoDB Connection Timeouts When Calling External APIs in AWS Amplify

0

I'm experiencing persistent MongoDB connection timeouts in my Next.js application deployed on AWS Amplify. The connection errors specifically occur when my application attempts to call external APIs like Gemini AI. Looking at the logs:

The application starts normally (Next.js 14.2.28 initializes) When attempting to call external APIs, MongoDB connections time out with error: MongoServerSelectionError: Socket 'secureConnect' timed out after 29329ms (connectTimeoutMS: 20000)

These timeouts occur immediately before or during external API calls

Important: The application works perfectly fine in my local environment. This issue only occurs in AWS Amplify. Environment Details

  • AWS Amplify (not direct Lambda)
  • Memory: 1024 MB (Max used: ~200 MB)
  • MongoDB Atlas cluster
  • Next.js 14.2.4
  • MongoDB driver version: 6.16
  • Mongoose: 8.12.2
  • Public VPCs and subnets

What I've Tried

  • Increased the MongoDB connection timeout (currently 20000ms)
  • Tested with a higher tier MongoDB Atlas instance (M10) with the same results
  • Successfully resolved some API issues by using aws-sdk for those calls
  • Been working on this issue for 15 days
  • MongoDB team confirmed I need to check with AWS

Questions

Any insights or recommendations would be greatly appreciated!

asked 11 days ago20 views
1 Answer
0

The MongoDB connection timeouts you're experiencing in AWS Amplify, specifically when calling external APIs like Gemini AI, could be caused by several factors. Here are some insights and recommendations:

  1. Network Configuration: Even with public VPCs, there might be network configuration issues preventing proper outbound connections. Ensure that your Amplify environment has the necessary network access to both MongoDB Atlas and the external APIs you're calling.

  2. Connection Pooling: The issue might be related to how connections are managed. Implement connection pooling for MongoDB to reuse connections efficiently, especially when making external API calls that may cause delays.

  3. Asynchronous Operations: Structure your code to handle MongoDB operations and external API calls asynchronously. This can help prevent blocking operations that might lead to timeouts.

  4. Timeout Settings: While you've increased the MongoDB connection timeout, also consider adjusting other timeout settings, including those for the external API calls and any intermediate network components.

  5. Resource Constraints: Although your memory usage seems low, there might be other resource constraints. Monitor CPU usage and network throughput to ensure they're not bottlenecks.

  6. AWS SDK for External APIs: For calling the Gemini API using AWS SDK, you can use the AWS SDK for JavaScript and the AWS.HttpClient class. Here's a basic example:

    const AWS = require('aws-sdk');
    const httpClient = new AWS.HttpClient();
    
    const url = 'https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent';
    const params = {
      key: 'YOUR_GEMINI_API_KEY'
    };
    const body = JSON.stringify({
      contents: [{
        parts: [{ text: "Explain how AI works" }]
      }]
    });
    
    const request = new AWS.HttpRequest(url, 'us-east-1');
    request.method = 'POST';
    request.headers['Content-Type'] = 'application/json';
    request.body = body;
    
    httpClient.handleRequest(request, params, (response) => {
      let responseBody = '';
      response.on('data', (chunk) => {
        responseBody += chunk;
      });
      response.on('end', () => {
        console.log(responseBody);
      });
    }, (error) => {
      console.error('Error:', error);
    });
  7. MongoDB Atlas Configuration: Double-check your MongoDB Atlas network settings. Ensure that the IP ranges for your AWS Amplify environment are whitelisted in MongoDB Atlas.

  8. Monitoring and Logging: Implement detailed logging for both MongoDB operations and external API calls. This can help identify exactly where and when the timeouts occur.

  9. Connection Warm-up: Implement a warm-up process for your MongoDB connections, especially after periods of inactivity.

  10. AWS Amplify Specific Settings: Review your Amplify configuration, particularly any settings related to network access or function timeouts.

If the issue persists after trying these solutions, consider reaching out to AWS Support for a more in-depth investigation of your specific Amplify environment and its interaction with external services.
Sources
REL05-BP05 Set client timeouts - AWS Well-Architected Framework (2023-04-10)
Lambda's Provision Concurrency For Low Latency | AWS re:Post
Next.js App Deployed with Elastic Beanstalk Unable to Connect to MongoDB Atlas | AWS re:Post

profile picture
answered 11 days ago
  • I have tried these generic things already. Nothing is working here. I tried the aws-sdk code shared above, getting below error,

    [AI] Error for <email>: error:1E08010C:DECODER routines::unsupported [AI] Error stack: Error: error:1E08010C:DECODER routines::unsupported at setKey (node:internal/tls/secure-context:93:11) at configSecureContext (node:internal/tls/secure-context:204:7) at Object.createSecureContext (node:_tls_common:116:3) at Object.connect (node:_tls_wrap:1763:48) at Agent.createConnection (node:https:171:22) at Agent.createSocket (node:_http_agent:340:26) at Agent.addRequest (node:_http_agent:288:10) at new ClientRequest (node:_http_client:334:16) at Object.request (node:https:379:10) at features.constructor.handleRequest (webpack-internal:///(rsc)/./node_modules/aws-sdk/lib/http/node.js:46:23) at eval (webpack-internal:///(rsc)/./lib/ai/email-generator.jsx:67:24) at new Promise (<anonymous>) at generateAIEmail (webpack-internal:///(rsc)/./lib/ai/email-generator.jsx:60:22) at createEmailContent (webpack-internal:///(rsc)/./lib/email/personalization.jsx:32:113) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async eval (webpack-internal:///(rsc)/./lib/email/campaign-processor.jsx:72:32) at async processContact (webpack-internal:///(rsc)/./path/to/file/campaign-processor.jsx:116:24) at async Promise.allSettled (index 0) at async runCampaign (webpack

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions