Unable to Connect to Amazon Bedrock LLMs from EKS-hosted Application (Connect Timeout)

Question

We have been successfully invoking LLMs and embedding models through Amazon Bedrock from an application hosted in Amazon EKS (us-west-1 region) for the past 5-6 days. However, since yesterday, we have been consistently encountering a connect timeout error while trying to hit the LLMs. The error message is as follows:

`Connect timeout on endpoint URL: "https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.nova-lite-v1%3A0/converse"
`
Bedrock client is configured with adequate timeout and retry mechanisms, ensuring sufficient limits for the specified model. Below is a sample code snippet that reflects our current implementation:

```
import boto3
import json
client = boto3.client("bedrock-runtime", region_name="us-east-1")

message_list = []
initial_message = {"role": "user", "content": [{ "text": "What is the cosine of 7?" }]}

message_list.append(initial_message)

response = client.converse(
    modelId="amazon.nova-lite-v1:0",
    messages=message_list,
    inferenceConfig={"maxTokens": 2000, "temperature": 0},
    system=[{"text": "Helpful assistant."}]
)

print(f"Call to bedrock model: {response}")

```
Could you please investigate what might have changed in the past few days that could be causing these timeouts? Let us know if any recent updates or service issues might be impacting the connectivity.

Additional Context:
* The issue is persistent and affects all requests to the LLM models.
* The same application and configuration worked without any issues before.
* No changes were made to our network settings, IAM policies, or application code.

Looking forward to your assistance in resolving this.

re:Post Agent · Answer

The connect timeout error you're experiencing suggests there might be a network connectivity issue between your EKS cluster in us-west-1 and the Bedrock endpoint in us-east-1. This could be caused by several factors:

1. Network Configuration: There might have been changes to the VPC, security groups, or network ACLs that are now blocking outbound traffic to the Bedrock endpoint.

2. DNS Resolution: Ensure that DNS resolution is working correctly within your EKS cluster, as the error suggests it's unable to resolve or connect to the Bedrock endpoint.

3. IAM Permissions: While you mentioned no changes to IAM policies, it's worth double-checking that the necessary permissions to access Bedrock are still in place and haven't been inadvertently modified.

4. Service Availability: Although unlikely, there could be a service issue with Bedrock in the us-east-1 region. Check the AWS Service Health Dashboard for any reported issues.

5. Cross-Region Latency: The fact that you're calling Bedrock in us-east-1 from us-west-1 introduces additional latency. Consider using a Bedrock endpoint in the same region as your EKS cluster if possible.

To troubleshoot this issue:

1. Verify network connectivity by attempting to ping or telnet to the Bedrock endpoint from within your EKS cluster.

2. Check your VPC's network configuration, including security groups and NACLs, to ensure outbound traffic to the Bedrock endpoint is allowed.

3. Review your IAM roles and policies to confirm they still have the necessary permissions to access Bedrock.

4. Try increasing the connect_timeout in your boto3 client configuration:

```python
   from botocore.config import Config
   config = Config(connect_timeout=30, retries={'max_attempts': 3})
   client = boto3.client("bedrock-runtime", region_name="us-east-1", config=config)
   ```

5. If possible, test the same code from an EC2 instance in the same VPC as your EKS cluster to isolate whether the issue is specific to EKS or affects the entire VPC.

6. Consider using AWS VPC endpoints for Bedrock to improve connectivity and reduce latency.

If the issue persists after these steps, it would be advisable to contact AWS support for further investigation, as there might be underlying service or network issues that are not immediately apparent.

**Sources**  
[Solving 'Read Timed Out' Error and High Latencies in Amazon Bedrock with AWS Java SDK Client | AWS re:Post](https://repost.aws/articles/AR_A_6aqweQYOR5FCgvM2vdA/solving-read-timed-out-error-and-high-latencies-in-amazon-bedrock-with-aws-java-sdk-client)  
[Unexpected Delay in Amazon Bedrock Response | AWS re:Post](https://repost.aws/questions/QU7emYZMABQIS7-FD0rw9Ljg/unexpected-delay-in-amazon-bedrock-response)  
[Prevent read timeouts in Amazon Bedrock | AWS re:Post](https://repost.aws/knowledge-center/bedrock-large-model-read-timeouts)

Vivek V [AWS Hero] · Answer

You can verify active inference profiles and potentially resolve the connection timeout issue, you should:

1. List the inference profiles in the us-east-1 region using the AWS CLI command:

```bash
aws bedrock list-inference-profiles --region us-east-1
```

2. If you see an output similar to the one you provided, with an active inference profile like:

```json
{
  "inferenceProfileArn": "arn:aws:bedrock:us-east-1:123456789012:inference-profile/us.amazon.nova-lite-v1:0",
  "models": [
    {
      "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-lite-v1:0"
    },
    {
      "modelArn": "arn:aws:bedrock:us-west-2::foundation-model/amazon.nova-lite-v1:0"
    },
    {
      "modelArn": "arn:aws:bedrock:us-east-2::foundation-model/amazon.nova-lite-v1:0"
    }
  ],
  "inferenceProfileId": "us.amazon.nova-lite-v1:0",
  "status": "ACTIVE"
}
```

3. Then use the `inferenceProfileId` as the model ID in your code. In this case, you would use "us.amazon.nova-lite-v1:0" as the model ID.

4. Modify your code to use this inference profile ID:

```python
client = boto3.client("bedrock-runtime", region_name="us-east-1")

response = client.converse(
    modelId="us.amazon.nova-lite-v1:0",  # Use the inference profile ID here
    messages=message_list,
    inferenceConfig={"maxTokens": 2000, "temperature": 0},
    system=[{"text": "Helpful assistant."}]
)
```

This approach should help resolve the connection timeout issue by using the correct inference profile for cross-region model access...

Unable to Connect to Amazon Bedrock LLMs from EKS-hosted Application (Connect Timeout)

Add your answer

Relevant content