OSError: Tunnel connection failed: 403 Forbidden

Hi vamshi,

Let's dive right in!

Clarifying the Issue

You'd like to resolve the OSError: Tunnel connection failed: 403 Forbidden issue when working with Amazon SageMaker and MLflow.

---

Our Recommended Solution

The error 403 Forbidden combined with ProxyError usually points to a proxy configuration issue when making API requests to SageMaker endpoints. Here are actionable steps to troubleshoot and resolve this issue:

1. Verify Network Proxy Configuration

• Ensure that your environment's proxy settings are correctly configured. The HTTPS_PROXY and HTTP_PROXY environment variables need to be set properly.
• Check your proxy server to confirm that it allows outbound connections to the SageMaker endpoint:
  https://eu-west-1.experiments.sagemaker.aws.

Example (for CLI or environment setup):

export HTTPS_PROXY=http://proxy-server:port
export HTTP_PROXY=http://proxy-server:port

• If you're using a corporate network, consult your network administrator to confirm access permissions.

2. Add SageMaker Endpoints to the Trusted List

• Ensure that your proxy/firewall allows access to the Amazon SageMaker domain and endpoint for your AWS Region. In this case, you need to add the following endpoints to your trusted list:
  https://*.sagemaker.aws and eu-west-1 endpoints specifically.

3. AWS IAM Permissions

• Confirm that the IAM role or user accessing SageMaker has sufficient permissions. The error can sometimes surface when authorization to SageMaker resources is denied.
• Attach the necessary permissions for mlflow integration with SageMaker. At minimum:

  {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "sagemaker:CreateExperiment",
          "sagemaker:CreateTrial",
          "sagemaker:CreateTrialComponent",
          "sagemaker:List*",
          "sagemaker:Get*"
        ],
        "Resource": "*"
      }
    ]
  }

4. Disable Proxy for Internal AWS Services (Optional)

If SageMaker is being accessed internally within AWS services, bypass the proxy for AWS internal endpoints:

export NO_PROXY=amazonaws.com,*.sagemaker.aws

5. Verify MLflow Client Configuration

• Double-check your MLflow configuration and tracking_uri. If you’re using SageMaker with MLflow, ensure the correct endpoint and credentials are passed:

import mlflow

mlflow.set_tracking_uri("https://<region>.experiments.sagemaker.aws")

---

If you've checked these steps and the error persists, please reach out to us for further troubleshooting.

Cheers! Aaron 😊