- 最新
- 投票最多
- 评论最多
Athena's query processing naturally triggers multiple Lambda functions (around 40) because it splits your query into parallel tasks for faster processing. This high number of Lambda invocations typically occurs when queries need to scan large amounts of data.
But please note Athena is a serverless service, which means that all the underlying computing resources are managed by AWS so there is no infrastructure to set up or manage. It is not possible to force Athena to use just one or two Lambdas (as parallel processing is fundamental to its performance).
The error you're experiencing is due to your Athena federated query exhausting the available RDS connections. This happens because Athena is invoking multiple Lambda functions concurrently to process your query, and each Lambda function is establishing connections to your RDS database.
Here's what's happening:
-
Athena federated queries work by invoking Lambda functions to connect to external data sources like your RDS database.
-
For complex queries with multiple joins and operations, Athena may invoke many Lambda functions concurrently (in your case, around 45).
-
Each Lambda function creates its own connection to the RDS database, quickly exhausting your available connection slots.
-
The error message indicates that all remaining connection slots are reserved for special users (non-replication superuser, rds_reserved, and rds_superuser).
To resolve this issue, you have several options:
-
Implement RDS Proxy between your Lambda functions and RDS database. RDS Proxy manages a pool of shared database connections, which enables your Lambda functions to reach high concurrency levels without exhausting database connections. This is recommended by AWS for production environments with many concurrent connections.
-
Optimize your query to reduce complexity. Your query contains multiple joins, CTEs, and complex operations that might be causing Athena to split the work across many Lambda functions. Try simplifying the query or breaking it into smaller parts.
-
Increase the maximum connections allowed in your RDS instance, if possible. However, this is more of a workaround than a solution.
-
Consider implementing connection pooling in your Lambda function code if you're not using RDS Proxy.
-
Break down large queries into smaller, more manageable parts that can be executed sequentially rather than all at once.
The most effective solution would likely be implementing RDS Proxy, as it's specifically designed to handle this type of scenario where many Lambda functions need to connect to an RDS database concurrently.
Sources
Athena Federated Query - MySQL - Lambda timeout troubleshooting | AWS re:Post
Which is preferred, Lambda->RDS Proxy->RDS or Lambda->RDS? | AWS re:Post

My question is, why is around 40 lambda invoked for one query. I understand that we can add an RDS proxy but I want to know if Athena can optimize this where maybe one or two lambdas are executed.