Dispatch error (timeout) when retrieving secrets from SecretsManager - Rust SDK

Question

Most of my info is in this github discussion thread.

Essentially, I'm using the boilerplate code to fetch secrets with aws-sdk-secretsmanager (aws-sdk-rust) I get `Could not retrieve secret 'arn:aws:secretsmanager:eu-west-1:1234512345:secret:MySecret': DispatchFailure(DispatchFailure { source: ConnectorError { kind: Timeout, source: hyper::Error(Connect, HttpTimeoutError { kind: "HTTP connect", duration: 3.1s }), connection: Unknown } })`

Permissions are as open as possible. 100% fail rate. Useless error message. Other log lines say things like: `DEBUG Lambda runtime invoke{requestId="1b61cb28-13c0-49c7-a952-a53508cfbc23" xrayTraceId="Root=1-682da77b-6d2a38f96ccd11e9239ef956;Parent=6442f41ea2a4b175;Sampled=0;Lineage=1:0546552b:0"}:Secrets Manager.GetSecretValue{rpc.service="Secrets Manager" rpc.method="GetSecretValue" sdk_invocation_id=3492809 rpc.system="aws-api"}:try_op:try_attempt{attempt=1}: current throughput: 0 B/s is below minimum: 1 B/s`. 0 B/s seems like a red flag that has me considering filing this as a bug.

My code:
```rust
// Start of function..
let shared_config = aws_config::defaults(BehaviorVersion::v2025_01_17())
      .region("eu-west-1")
      .load()
      .await;

let secret_string = match resp.secret_string() {
        Some(s) => s,
        None => {
            return Err(format!("Missing secret '{name}' in SecretsManager"));
        }
};

let rds_secret = match serde_json::from_str::(secret_string) {
        Ok(s) => s,
        Err(e) => {
             error!("Error parsing secret '{}': {}", name, e);
              return Err(format!("Parsing secret error"));
        }
 // Some more code & end of function      
```

rePost-User-8823240 · Accepted Answer

Solved this by moving the lambda out of the VPC. I first tried setting up the endpoint for secretsmanager, with a fully open security group, but this didn't work either.

rePost-User-8823240 · Answer

Can't get XRay traces to work. Get error messages `Role AROASDFTK3FXTEUSIJVRM trusts too many services, expected only 1.` but this role does not exist in my IAM..

re:Post Agent · Answer

The issue you're experiencing with retrieving secrets from AWS Secrets Manager using the AWS SDK for Rust appears to be related to network connectivity or configuration problems. The error message indicates a dispatch failure due to a timeout when trying to connect to the Secrets Manager service.

Here are a few things to check and try:

1. VPC Configuration: If your Lambda function is running in a VPC, ensure that it has proper network connectivity to reach AWS services. This typically involves setting up a NAT Gateway or VPC endpoints for Secrets Manager.

2. Security Groups and Network ACLs: Verify that the security groups and network ACLs associated with your Lambda function allow outbound traffic to the Secrets Manager service.

3. DNS Resolution: Make sure that DNS resolution is working correctly within your VPC. The error message suggests that the service endpoint couldn't be reached, which could be due to DNS issues.

4. Timeouts: The default timeout for the SDK might be too short. You could try increasing the timeout in your SDK configuration.

5. Region Configuration: Double-check that the region you're specifying ("eu-west-1") is correct and matches the region where your secret is stored.

6. IAM Permissions: Although you mentioned permissions are as open as possible, it's worth double-checking that the IAM role associated with your Lambda function has the necessary permissions to access Secrets Manager.

7. VPC Endpoints: If you're using VPC endpoints for Secrets Manager, ensure they are correctly configured and associated with the right subnets.

The debug log showing "0 B/s" throughput is indeed concerning and suggests that no data is being transferred. This could be due to network connectivity issues or potentially a bug in the SDK.

If none of these suggestions resolve the issue, you may want to consider:

1. Testing with a different SDK version or even a different programming language to isolate whether it's a Rust SDK-specific issue.

2. Using AWS X-Ray for more detailed tracing of the request to pinpoint where exactly the timeout is occurring.

3. Reaching out to AWS support or filing a bug report with the AWS SDK for Rust team if you suspect it might be a bug in the SDK itself.  
**Sources**  
[Handling errors in the AWS SDK for Rust - AWS SDK for Rust](https://docs.aws.amazon.com/sdk-for-rust/latest/dg/error-handling.html)  
[AWS Glue Secrets Manager - Network unreachable | AWS re:Post](https://repost.aws/questions/QUUn_c2lGbRvSd93407IK8LA/aws-glue-secrets-manager-network-unreachable)

Dispatch error (timeout) when retrieving secrets from SecretsManager - Rust SDK

Add your answer

Relevant content