Lambda to DB connectivity - best practices

1

Hi,

I have a some AWS Lambda functions which connect to a Postgres SQL database hosted on EC2 and and Oracle DB hosted out of AWS. I have been facing some intermittent connectivity challenges in connecting to DB hosted outside AWS.

Also, basis some research I came across articles stating that directly connecting Lambda to a DB is not ideal for production workloads as Lambda cant maintain a connection pool. There is an option for AWS RDS proxy to handle this but that appears to be only working for MYSQL hosted on RDS.

Any pointers on best practices to connect Lambda to a relational database both within and outside AWS network?

Regards, dbeings

asked 2 years ago15.3K views
3 Answers
2

First, regarding the connectivity. I assume your self hosted database is running in a private subnet in a VPC. You will need to attach your Lambda functions to that VPC and allow the connectivity inside the VPC using Security Groups. With regards to the database hosted on premises, I assume it is not exposed to the internet, and as such, you should have a direct connect (or VPN) from your network to a VPC in AWS. In this case you will also need to attach the function to the VPC and use routing and security groups to allow the connectivity.

The main issue with connecting Lambda functions to a relational database is the number of connections. As Lambda functions can scale very high, very fast, it may cause a lot of connections to the database, which may cause issues on the DB. For that reason we recommend using RDS proxy, which is not an option in your case. The other option is to limit the concurrency if your functions to a number which is supported by your database. How to do that depends on the function's event source. For instance, if the function is invoked from API Gateway, you can use throttling in the gateway. If the function is invoked from SQS, you can use the event source's Max Concurrency setting. You can always use Reserved Concurrency on the function to limit its maximum concurrency.

Finally, I would recommend that you create the connection outside the function handler, and reuse it across all invocations. That way you reduce the create connection/close connection load on the database, and you reduce your function's duration, which will also reduce your cost.

profile pictureAWS
EXPERT
answered 2 years ago
1

Using AWS Lambda to connect to a database, both within and outside of the AWS network, can be challenging due to the stateless and short-lived nature of Lambda functions. Here are a few best practices to consider when connecting Lambda to a relational database:

Use a connection pooling library: Since Lambda functions cannot maintain a persistent connection to a database, it's best to use a connection pooling library to manage database connections. This can help reduce the overhead of creating and closing connections for each Lambda invocation.

Use a VPC endpoint: If the database is hosted within the same VPC as your Lambda function, you can use a VPC endpoint to connect to the database. This can help improve security and reduce the risk of connection interruptions.

Use an API Gateway: You can use an API Gateway to create an HTTP(s) endpoint that your Lambda function can connect to. The API Gateway can then forward requests to the appropriate database. This can help improve security and scalability, and also provide a level of abstraction between your Lambda function and the database.

Use AWS RDS proxy: AWS RDS proxy can handle the connection pooling and also improve security by providing a central point of authentication and access control to your RDS instances.

Use a Data Migration Services: If you are facing connectivity challenges with databases hosted outside AWS, consider using a data migration service like AWS DMS (Database Migration Service) to replicate data to a database within the AWS network.

Evaluate the use of AWS AppSync: AppSync is an AWS service that allows you to develop GraphQL APIs, it also allows you to connect your backend to multiple data sources like RDS, DynamoDB, Elasticsearch, and more. It can also cache the queries to improve performance.

Please note that, AWS RDS proxy supports only MYSQL hosted on RDS. Please also note that, in most cases, it is recommended to use a managed service like RDS, Aurora, or DocumentDB, instead of running your own databases on EC2 instances.

answered 2 years ago
0

Small clarification:

Amazon RDS Proxy is available for Amazon Aurora with MySQL compatibility, Amazon Aurora with PostgreSQL compatibility, Amazon RDS for MariaDB, Amazon RDS for MySQL, Amazon RDS for PostgreSQL, and Amazon RDS for SQL Server.

https://aws.amazon.com/rds/proxy/

AWS
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions