AWS Lambda using NET8 image hangs on database access

Question

We moved to NET8, so therefor we started using a NET8 image for a Lambda.
The Lambda accesses two MySql databases in RDS.
On the Lambda we see in Configuration -  RDS Databases that both databases should be accessible with the provided VPC's and security groups.
But the problem is that we can only access one of them. The moment we reference the other database the Lambda hangs until the timeout is triggered.
There is no error/exception whatsoever.
We are using Entity Framework Core 8.0 with Pomelo 8-beta2
Both databases are 5.7, 
For both connectionstrings => ServerVersion.AutoDetect(connectionString) returns "5.7.12-mysql"
In the Pomelo code I can see that it actually does a connection.Open() for the AutoDetect, so that must mean it can access the database there right?

in C# 
database1Context.Database.CanConnect() works
database2Context.Database.CanConnect() hangs

var query = database1Context.EntitySet1.Where( x => x.Id ==1)   works
var query = database2Context.EntitySet1.Where( x => x.Id ==1)   hangs

And these last ones only build the query, they don't even actually execute it yet.

The Lambda was working fine when using NET6 (without an image).
And we are using the same VPC and securitygroups as with the NET6 version.

We tried adding Pipelining=False to the connectionstring, no difference.

I really need some help here on how to attack the problem. How can I find out what is wrong?
Why can it access one database and not the other one?
Highly frustrating the lack of errors. Been trying all kind of things for two full days. Any hints appreciated!

Answer

Some other things to consider:
* Is it always the second call that hangs or always database2Context? What if you switch the order you connect to 2 then 1?
* What are the differences between DB 1 and DB 2? 
* Have you done anything custom or non-default with certificates? 
* If you increase the Lambda timeout to the max 15 minutes, maybe it will give the .NET code enough time to hit it's own timeout so that at least you can get a stack trace.

For what it's worth, I was able to connect to 2 different RDS MySQL databases on a VPC using the .NET 8 image Lambda using `MySqlConnector.MySqlConnection` (I don't have any experience with Pomelo unfortunately). The MySQL version was v5.7.37, which is the oldest allowed in the dropdown.

Answer

Hangs are always frustrating without actionable errors. Are you able to change some experimental variables to eliminate or narrow down the issue? For example:
* Does it also happen with a .NET 6 container image Lambda?
* Does it happen with .NET 8 outside of Lambda? (can you run this from an EC2 instance or development machine?)
* Does it still work fine with .NET 6 Lambda?
* Are you using native AOT?
* Would it make sense to also open an issue here: https://github.com/PomeloFoundation/Pomelo.EntityFrameworkCore.MySql/issues ?

Answer

Are you using `public.ecr.aws/lambda/dotnet:8` or `public.ecr.aws/lambda/dotnet:8-preview` ? Or something else for the container image base?

Answer

I upgraded the EntityFramework packages to 8.0.2 (were 8.0.1) and Pomelo to 8.0.0 (was beta2) and now it is no longer hanging. Don't know which packages were the cause. I may try to find out.

Answer

Going back to the beta version (just for testing) did not reintroduce the problem. So we have no clue what was the actual cause of the problem

AWS Lambda using NET8 image hangs on database access

Relevant content