By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post

Old Lambda functions using .net2.1 and .net3.1 and connecting to MySQL RDS got broken

0

Hello, we have a huge problem with lots of old Lambda functions.

Last month some old Lambdas (.net2.1, .net3.1) stopped working in one of our accounts.

There was a strange error

{ "errorType": "KeyNotFoundException", "errorMessage": "The given key '0' was not present in the dictionary.", "stackTrace": [ "at System.Collections.Generic.Dictionary`2.get_Item(TKey key)", "at MySql.Data.MySqlClient.MySqlField.SetFieldEncoding()", "at MySql.Data.MySqlClient.NativeDriver.GetColumnData(MySqlField field)", "at MySql.Data.MySqlClient.NativeDriver.GetColumnsData(MySqlField[] columns)", "at MySql.Data.MySqlClient.Driver.GetColumns(Int32 count)", "at MySql.Data.MySqlClient.ResultSet.LoadColumns(Int32 numCols)", "at MySql.Data.MySqlClient.ResultSet..ctor(Driver d, Int32 statementId, Int32 numCols)", "at MySql.Data.MySqlClient.Driver.NextResult(Int32 statementId, Boolean force)", "at MySql.Data.MySqlClient.MySqlDataReader.NextResult()", "at MySql.Data.MySqlClient.MySqlDataReader.Close()", "at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()", "at MySql.Data.MySqlClient.Driver.Configure(MySqlConnection connection)", "at MySql.Data.MySqlClient.MySqlConnection.Open()", "at Core.MySql_class..ctor<.....>", "at <.....>", "at lambda_method(Closure , Stream , Stream , LambdaContextInternal )" ] }

We have updated to .net6.0 and the problem disappeared without any changes in db settings or function code.

Now we have the same problem in another account (and another RDS MySQL), and we don't understand why and how could functions that have worked for some years could get broken without any changes.

Does anyone have any ideas of fixing it without manual updating functions (about 500 in total) or preventing the problem?

  • Karina
    8 months ago

    As I think, the problem is that MySQL returns some wrong result, and the error appers when it is parsed to my MySQL result data object. But I can't figure out what the problem is, becfause when I update the function to .net6.0 (I need to do this. because I can;t apply changes to Lambda .net 2.1 and 3.1) it starts working

  • secondabhi_aws EXPERT
    8 months ago

    Are you looking to change runtime version of lambda function?

  • Karina
    8 months ago

    I know how to update lambda functions, but the problem is that now we have a huge amount of functions that stopped working accidently. And I want to know what is the problem related with and how to prevent it's appearance. I think it is related with DB connection, but we didn;t change anything in RDS MySQL instances or their configuration.

  • Karina
    8 months ago

    One more notice: I have a feeling that problem only appears when we are parsing the MySQL Query Result into List<Dictionary<string, string>> . May the problem in old Lambda functions be related to using System.Collection.Generic? Can't it somehow become broken for older versions of .net Lambda?

  • Karina
    8 months ago

    @secondabhi_aws , is it possible to update runtime for .net 2.1 and 3.1 Lambda without deleting ang re-publishing functions? Or may be do something else to return them to working state

Topics
ServerlessCompute.NET on AWSDatabase
Tags
AWS Lambda.NET on AWSMySQL
Language
English
Karina
asked 8 months ago221 views
1 Answer
0

Hi Karina,

I'm not exactly sure, if this is the exact ask but based on your description, I'm assuming you want to update runtime version of those lambda functions in bulk:

I'd suggest you to see documentation here for Lambda Automatic, Function, Manual update configuration guidance: https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html

There is a general purpose script for Python 3.6 to 3.9 function updates with same warnings as above: Update runtime version from python3.6 to 3.9. This script can be used for other languages as well

As stated above, it's always best practice to evaluate lambda runtimes, deprecation/support deadline, release notes and prepare an upgrade path than to forcefully update runtime versions.

Another way of doing same is:

  1. List All lambda functions ARN and their run time

       aws lambda list-functions --profile <cli_profile> --query "Functions[*].[FunctionArn,Runtime]" --output table

  2. Paste that output in the spreadsheet, in each row add another column for to-be-new-run-time and then create CLI command by concatenating fixed part of next command, function arn and to-be-new-run-time:

       aws lambda update-function-configuration --function-name <function_arn_from_spreadsheet> --runtime <to_be_new_run_time> --profile <cli_profile>

Here is how it'd look like.

Enter image description here

Formula: =CONCATENATE("aws lambda update-function-configuration --function-name ",A2, C2," --profile <cli_profile>")

Hope you find this helpful.

Comment here if you have additional questions. Happy to help.

profile pictureAWS
EXPERT
secondabhi_aws
answered 8 months ago
  • Karina
    8 months ago

    Hi, Honestly I'm just confused because of the situation itself. I have thought that Lambda functions are independent and isolated, so if the function works (for example, gets some data from db) - it will work until something will change (for example, db connection credentials). And if the DB is not changed, updated, modified, db user is not changed - the function should work. But for now I understand that there are some circumstances that can cause hundreds of lambdas got broken without any evident reasons. And I am really in panic, because I don't understand why and when it can happen.

  • Karina
    8 months ago

    And the question according to your answer - is it possible to update lambda runtime from 2.1 now? Or only delete function and upload it again?

  • secondabhi_aws EXPERT
    8 months ago

    I'd suggest you to try it for one and test the behavior. This is why I mentioned in the beginning of my answer that we should follow the configuration guidance and keep working periodic updates specially for IaC. AWS notifies for version deprecation. Also, your understanding is right but there can be releases which may need some updates in your code such as you are using certain feature in .net 2.1 but that's now deprecated in newer version, so lambda function wouldn't work the way you'd expect, but that had nothing to do with lambda but runtime version. It's little difficult to say, without seeing the code, what would have caused but since you are in situation, where upgrade is the only option, I'd suggest you to do it for few lambda functions and test and fix the error.

    Based on documentation, you may not be able to update the runtime version now but I'd give a try through CLI and see how that goes. Otherwise last resort is redeployment.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions

Relevant content