What is the best way to cache database connection and retrieve secrets securely in lambda function?

Question

Hello all,

I would like to be able to cache some static code and also retrieve secrets securely in lambda function.

Attempt #1 - Pass secrets as env variables: This allowed me to use execution environment caching where some of the static code only needs to be initialized once and might be used by the consecutive invocations. (I am aware that AWS doesn't provide any guarantees on execution environment caching yet it's still used and even some AWS articles is suggested)

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
   // read secrets from env and construct DB url
   let connection = PgPool::connect("...").await?;

    lambda_http::run(service_fn(|event: Request| login(event, &connection))).await?;
    Ok(())
}

The problem is that I have to pass secrets as env variables which is not sure and not recommend.

Attempt #2 - Read secrets from AWS Parameter and Secrets Lambda extension:

I added AWS Parameter and Secrets Lambda extension layer to my lambda function and this allowed me to retrieve secrets securely however now I have to construct my connection in every single invocation and I can't use execution environment caching.

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
    let http_client : reqwest::Client = reqwest::Client::new();
    lambda_http::run(service_fn(|event: Request| login(event, &http_client))).await?;
    Ok(())
}

#[instrument]
pub async fn login(
    event: Request,
    http_client: reqwest::Client,
) -> Result<impl IntoResponse, E> {
    let response = http_client.get("http://localhost:2773/secretsmanager/get?secretId=your-secret-id")
        .header("X-Aws-Parameters-Secrets-Token", env::var("AWS_SESSION_TOKEN").unwrap())
        .send()
        .await?
        .text()
        .await?;

    // parse "response" to extract database username and password and then create database connection every-time lambda is invoked.
    // at this point only thing that is cached is retrieving secret from secret manager thanks to AWS Parameter and Secrets Lambda extension layer.
    let connection = PgPool::connect("...").await?;

    Ok(Response::builder()
        .status(200)
        .body("Ok".toOwned())
        .unwrap())
}

For every single invocation, I have to construct database connection.

I naively tried to move

let response = http_client.get("http://localhost:2773/secretsmanager/get?secretId=your-secret-id")
        .header("X-Aws-Parameters-Secrets-Token", env::var("AWS_SESSION_TOKEN").unwrap())
        .send()
        .await?
        .text()
        .await?;

to outside of the handler function but to my understanding, at that stage extension layer is not ready so call to http://localhost:2773/secretsmanager/get responded with not ready to serve traffic, please wait. I would love to know if it's possible to resolve this somehow.

Attempt #3 - Use IAM authentication to access RDS:

To my knowledge, aws-rust-sdk under development and at the moment there is no equivalent of rds-signer https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_rds_signer.html but let's assume there is one then I could do something like this:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
    // Assume that this exists.
    let signer = RdsSigner(...)
    // This should create a token which I can use to connect RDS
    let token = signer.getAuthToken()/await?;
    // at this point I can use `token` to create database connection
    let connection = PgPool::connect("...").await?; 
    lambda_http::run(service_fn(|event: Request| login(event, &connection))).await?;
    Ok(())
}

I can't test this since `RdsSigner` doesn't exist but it could work. However, the problem is the combination of these two facts;
- `token` generated by RdsSigner is valid for 15 minutes.
- Execution environment caching doesn't provide any guarantees and it could be longer than 15 minutes.

At this point I have to invalidate `token` variable after 15 minutes somehow. Even-then, I don't think changes made to variables outside of the handler function is visible to consecutive invocations. It looks like this would also wouldn't work. I have to switch to connection-per-invocation like I did in Attempt #2

Attempt #4 - RDS Proxy:

One could argue that I should be using RDS Proxy and not create "database connection" per lambda invocation. I do agree with this, however, this doesn't change the fact that I still need to initiate connection instances (construct connection object and connect to RDS Proxy etc) to RDS Proxy every-time. I still can't use execution environment caching due to the problems I mentioned in Attempt #2 and Attempt #3 because from lambda function's perspective only thing that changes is the database_host, how you retrieve the secrets etc is unrelated to whether you use RDS Proxy or not. I also hate the fact that RDS Proxy comes with cost so I would like to stay away from it if possible!

Just to be clear, RDS Proxy is about managing number of connections to database instance. What I am trying to address is to avoid creating connection instances in my program every-time lambda is invoked .

To reiterate, is there a way to retrieve secrets securely and avoid creating connections instances on every lambda invocation?

Disclaimer: I haven't bench-marked anything so I don't know which of the following is true:

It could be the cases that creating connections instances in every lambda invocation is so small that it could be ignored
It could be that it's actually takes few ms so worth avoiding

Answer

Lambda runs in execution environments (micro VMs). Each such EE, runs a single request at a time. When the function returns we keep the EE live for a few minutes so that if another request is received for the same function, we can invoke it directly without going through the cold start again. This means that we absolutely encourage customers to cache values between invocations (when it makes sense) by saving them in global/instance variables. Everything you save in a global variable will be accessible in the following invocations.

As you can cache values (and database connections) between invocations, it is recommended that you initialize those variables in the initialization phase of your function. This means, outside the handler code. Look at [this example](https://github.com/awslabs/aws-lambda-rust-runtime/blob/main/examples/basic-sdk/src/main.rs) and you will see that the S3 client is initialized in the main, and not in the handler. This means it will be called only once. (not an expert in Rust, so there may be other ways to achieve the same).

The recommendation with regards to secrets is to have environment variables with the secret name (in SSM/Secrets manager) and either use the extension or just read them at the function's initialization code. Save the credentials in the global variables and you can reuse them in all invocations. Same is true for database connections. Create the connection outside the handler, save it in a global variable, and use it in all invocations.

Last point is regarding RDS Proxy. If there is a chance that you function will scale very high, it is highly recommended that you use the proxy. Otherwise, you may overwhelm the database. Even if you do not anticipate high scale, the proxy can help with: 1. credentials (see above), and 2. better database failure handling.

Answer

Hi @Uri thanks for your answer.

I think we agree on the dots (1-) cache values 2-) read secrets securely) but I it's still not clear to me how to connect those dots especially when using Parameter and Secrets Lambda extension layer.

As I mentioned above, you can't access to Parameter and Secrets Lambda extension layer running on localhost outside of the handler function. This would mean that you can't cache code that depends on secrets, do you agree?

That being said, one could simply fetch secrets manually using SDK and cache the secrets in global variables as you suggested (that's what people are/were doing before Parameter and Secrets Lambda extension anyway).

To my understanding, one of the selling point of Parameter and Secrets Lambda extension is that you don't need to pollute your code with aws-sdk just because you want to cache secrets. However, that doesn't seem to be working in practice for cases where you want to have global variable that depends on secrets which you need to fetch from Parameter and Secrets Lambda extension layer.

To clarify, following is something you CANT do with Parameter and Secrets Lambda extension layer:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
    let http_client : reqwest::Client = reqwest::Client::new();
    
    // Note that we are outside of the handler function. This request will result in "not ready to serve traffic, please wait" 
    // which means you can't access to secrets outside of the handler function using Parameter and Secrets Lambda extension layer
    let secret_response = http_client.get("http://localhost:2773/secretsmanager/get?secretId=your-secret-id")
         .header("X-Aws-Parameters-Secrets-Token", env::var("AWS_SESSION_TOKEN").unwrap())
         .send()
         .await?
         .text()
         .await?;
    
    // You CAN'T seem to do this! "http://localhost:2773/secretsmanager/get" doesn't work outside of the handler function.
    let global_variable_that_depends_on_some_secret = use_secret_to_generate_gloabl_variable(secret_response);

    lambda_http::run(service_fn(|event: Request| login(event, &http_client))).await?;
    Ok(())
}

^This made me confused because I expected this to work given the fact that Parameter and Secrets Lambda extension layer was trying to make the following pattern which is widely used easier:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync + 'static>> {
    let shared_config = aws_config::from_env().load().await;
    let client = Client::new(&shared_config);
    let secret_response = client.get_secret_value().secret_id(name).send().await?;
    // You CAN do this because at this point lambda already have everything it needs to access secret store etc.
    let global_variable_that_depends_on_some_secret = use_secret_to_generate_gloabl_variable(secret_response);

    lambda_http::run(service_fn(|event: Request| login(event, &http_client))).await?;
    Ok(())
}

Given the fact that you mentioned cache is for 5 minutes, this would mean that Attempt #3 - Use IAM authentication to access RDS would also work because it's also using SDK to retrieve temporary token which can be done outside of the handler function.

To conclude, I was aware of the pattern where you would fetch secrets using SDK and cache them in global variable and given the fact that Parameter and Secrets Lambda extension layer introduced to make this process easier (https://aws.amazon.com/blogs/compute/using-the-aws-parameter-and-secrets-lambda-extension-to-cache-parameters-and-secrets/ see 2nd paragraph) I expected it to just work but it turns out that it's not exactly the same (for some cases) which lead to some confusion on my side. As a result, I won't be using Parameter and Secrets Lambda extension layer since it doesn't work for my use case :)

What is the best way to cache database connection and retrieve secrets securely in lambda function?

Relevant content