Using AWS SDK Service-Specific Endpoints for VPC Endpoint Connectivity

Introduction

AWS SDKs provide built-in service-specific endpoint configuration that allows you to redirect API calls to custom endpoints, including VPC endpoints. This feature is particularly valuable when private DNS resolution is not available or cannot be extended to your environment, such as cross-VPC scenarios, on-premises integration, or multi-account architectures. While AWS VPC endpoints typically use private DNS to automatically redirect service calls, there are scenarios where private DNS resolution is not suitable or available:

• Mixed connectivity requirements: In shared environments where some applications need public endpoints while others require private endpoints, changing the default DNS resolution for a service like redshift.amazonaws.com affects all applications uniformly
• Application-specific routing: Different applications within the same environment may need to use different endpoints based on security, compliance, or performance requirements

Solution Overview

AWS SDKs provide native service-specific endpoint configuration that redirects API calls to custom endpoints without requiring DNS modifications. This built-in functionality automatically handles endpoint resolution across all supported programming languages and tools, including:

Python (Boto3):

import boto3

# SDK automatically uses AWS_ENDPOINT_URL_REDSHIFT if set
redshift_client = boto3.client('redshift')

# Or configure programmatically
redshift_client = boto3.client(
    'redshift',
    endpoint_url='https://vpce-123.redshift.us-east-1.vpce.amazonaws.com'
)

Java (AWS SDK v2):

// SDK automatically uses aws.endpointUrlRedshift system property
RedshiftClient client = RedshiftClient.create();

// Or configure programmatically
RedshiftClient client = RedshiftClient.builder()
    .endpointOverride(URI.create("https://vpce-123.redshift.us-east-1.vpce.amazonaws.com"))
    .build();

Node.js (AWS SDK v3):

import { RedshiftClient } from "@aws-sdk/client-redshift";

// SDK automatically uses AWS_ENDPOINT_URL_REDSHIFT if set
const client = new RedshiftClient({});

// Or configure programmatically
const client = new RedshiftClient({
    endpoint: "https://vpce-123.redshift.us-east-1.vpce.amazonaws.com"
});

AWS CLI:

# CLI automatically uses AWS_ENDPOINT_URL_REDSHIFT if set
aws redshift describe-clusters

# Or specify endpoint directly
aws redshift describe-clusters \
  --endpoint-url https://vpce-123.redshift.us-east-1.vpce.amazonaws.com

This feature supports VPC endpoints, local development environments, and third-party AWS-compatible services through standardized configuration methods.

AWS SDK Service-Specific Endpoint Configuration

Method 1: Service-Specific Environment Variables

AWS SDKs support service-specific endpoint configuration using standardized environment variables:

# Service-specific endpoint for Redshift
export AWS_ENDPOINT_URL_REDSHIFT=https://vpce-1234567890abcdef0-abcdefgh.redshift.us-east-1.vpce.amazonaws.com

# Service-specific endpoint for S3
export AWS_ENDPOINT_URL_S3=https://vpce-1234567890abcdef0-abcdefgh.s3.us-east-1.vpce.amazonaws.com

# Service-specific endpoint for DynamoDB
export AWS_ENDPOINT_URL_DYNAMODB=https://vpce-1234567890abcdef0-abcdefgh.dynamodb.us-east-1.vpce.amazonaws.com

The environment variable format is AWS_ENDPOINT_URL_<SERVICE> where <SERVICE> is the AWS service identifier in uppercase.

Method 2: Shared AWS Config File

Configure service-specific endpoints in your AWS config file (~/.aws/config):

[default]
region = us-east-1

[services redshift]
endpoint_url = https://vpce-1234567890abcdef0-abcdefgh.redshift.us-east-1.vpce.amazonaws.com

[services s3]
endpoint_url = https://vpce-1234567890abcdef0-abcdefgh.s3.us-east-1.vpce.amazonaws.com

[services dynamodb]
endpoint_url = https://vpce-1234567890abcdef0-abcdefgh.dynamodb.us-east-1.vpce.amazonaws.com

Method 3: JVM System Properties (Java/Kotlin)

For Java applications, use JVM system properties:

# Service-specific endpoint for Redshift
-Daws.endpointUrlRedshift=https://vpce-1234567890abcdef0-abcdefgh.redshift.us-east-1.vpce.amazonaws.com

# Service-specific endpoint for S3
-Daws.endpointUrlS3=https://vpce-1234567890abcdef0-abcdefgh.s3.us-east-1.vpce.amazonaws.com

Method 4: Global Endpoint Override

Set a global endpoint that affects all AWS services:

# Environment variable
export AWS_ENDPOINT_URL=https://localstack:4566

# AWS config file
[default]
endpoint_url = https://localstack:4566
region = us-east-1

# JVM system property
-Daws.endpointUrl=https://localstack:4566

Configuration Precedence

AWS SDKs follow a specific precedence order for endpoint configuration:

1. Explicit endpoint in code (highest priority)
2. Service-specific environment variables (AWS_ENDPOINT_URL_<SERVICE>)
3. Service-specific config file settings ([services <service>])
4. Global environment variable (AWS_ENDPOINT_URL)
5. Global config file setting (endpoint_url)
6. Default service endpoints (lowest priority)

Ignoring Configured Endpoints

You can disable all custom endpoint configurations using:

# Environment variable
export AWS_IGNORE_CONFIGURED_ENDPOINT_URLS=true

# AWS config file
[default]
ignore_configured_endpoint_urls = true

# JVM system property
-Daws.ignoreConfiguredEndpointUrls=true

Note: Explicit endpoints set in code or command-line parameters (like --endpoint-url) are always used regardless of this setting.

Implementation Examples

Example 1: Cross-VPC Redshift Access

Scenario: Application in VPC-A needs to access Redshift through VPC endpoint in VPC-B.

Environment Variable Approach:

export AWS_ENDPOINT_URL_REDSHIFT=https://vpce-1234567890abcdef0-abcdefgh.redshift.us-east-1.vpce.amazonaws.com

Config File Approach:

[default]
region = us-east-1

[services redshift]
endpoint_url = https://vpce-1234567890abcdef0-abcdefgh.redshift.us-east-1.vpce.amazonaws.com

Example 2: Multi-Service VPC Endpoint Configuration

Scenario: Application needs private connectivity to multiple AWS services.

# Configure multiple service-specific endpoints
export AWS_ENDPOINT_URL_REDSHIFT=https://vpce-redshift.us-east-1.vpce.amazonaws.com
export AWS_ENDPOINT_URL_S3=https://vpce-s3.us-east-1.vpce.amazonaws.com
export AWS_ENDPOINT_URL_DYNAMODB=https://vpce-dynamodb.us-east-1.vpce.amazonaws.com
export AWS_ENDPOINT_URL_LAMBDA=https://vpce-lambda.us-east-1.vpce.amazonaws.com

Example 3: Development vs Production Environments

Development (LocalStack):

# All AWS services redirect to LocalStack
export AWS_ENDPOINT_URL=http://localhost:4566

# Test with any AWS SDK - automatically uses LocalStack
python -c "import boto3; print(boto3.client('s3').list_buckets())"
aws s3 ls  # Automatically uses LocalStack endpoint

Production (VPC Endpoints):

# Service-specific VPC endpoints for production
export AWS_ENDPOINT_URL_REDSHIFT=https://vpce-prod-redshift.us-east-1.vpce.amazonaws.com
export AWS_ENDPOINT_URL_S3=https://vpce-prod-s3.us-east-1.vpce.amazonaws.com

# Applications automatically use VPC endpoints without code changes
java -jar my-app.jar  # Uses VPC endpoints automatically
python my_script.py   # Uses VPC endpoints automatically

Example 4: AWS CLI with Service-Specific Endpoints

Multiple Services with Different Endpoints:

# Configure different endpoints for different services
export AWS_ENDPOINT_URL_REDSHIFT=https://vpce-redshift.us-east-1.vpce.amazonaws.com
export AWS_ENDPOINT_URL_S3=https://vpce-s3.us-east-1.vpce.amazonaws.com
export AWS_ENDPOINT_URL_DYNAMODB=http://localhost:8000  # Local DynamoDB

# Each command automatically uses the correct endpoint
aws redshift describe-clusters        # Uses VPC endpoint
aws s3 ls                            # Uses VPC endpoint  
aws dynamodb list-tables             # Uses local endpoint

Profile-Based Configuration:

# ~/.aws/config
[profile vpc-prod]
region = us-east-1

[profile vpc-prod services redshift]
endpoint_url = https://vpce-redshift.us-east-1.vpce.amazonaws.com

[profile vpc-prod services s3]
endpoint_url = https://vpce-s3.us-east-1.vpce.amazonaws.com

# Use with AWS CLI
aws --profile vpc-prod redshift describe-clusters
aws --profile vpc-prod s3 ls

Benefits

Network Security: Traffic remains within AWS backbone, never traversing the public internet.

Compliance: Meets regulatory requirements for private connectivity without complex DNS configurations.

Flexibility: Works across different network architectures and deployment models.

Service Identifiers

AWS SDKs use specific service identifiers for endpoint configuration. Common service identifiers include:

Service	Environment Variable	Config File Section
Amazon Redshift	AWS_ENDPOINT_URL_REDSHIFT	[services redshift]
Amazon S3	AWS_ENDPOINT_URL_S3	[services s3]
Amazon DynamoDB	AWS_ENDPOINT_URL_DYNAMODB	[services dynamodb]
AWS Lambda	AWS_ENDPOINT_URL_LAMBDA	[services lambda]
Amazon RDS	AWS_ENDPOINT_URL_RDS	[services rds]
Amazon EC2	AWS_ENDPOINT_URL_EC2	[services ec2]
Amazon CloudWatch	AWS_ENDPOINT_URL_CLOUDWATCH	[services cloudwatch]
AWS IAM	AWS_ENDPOINT_URL_IAM	[services iam]

Best Practices

Environment-Specific Configuration: Use different VPC endpoints for development, staging, and production environments.

Centralized Configuration: Store VPC endpoint URLs in configuration management systems rather than hardcoding in applications.

Health Checks: Implement application-level health checks to verify VPC endpoint connectivity.

Fallback Strategy: Consider implementing fallback to public endpoints for disaster recovery scenarios.

Security Groups: Ensure security groups allow traffic to VPC endpoint network interfaces on port 443.

Monitoring: Set up CloudWatch metrics and alarms to monitor VPC endpoint usage and performance.

Limitations and Considerations

Regional Scope: VPC endpoints are region-specific; multi-region applications need endpoints in each region.

Service Coverage: Not all AWS services support VPC endpoints; check service documentation for availability.

DNS Caching: Some applications may cache DNS resolutions; endpoint overrides bypass this caching.

Third-Party Tools: Some third-party tools may not respect AWS SDK endpoint overrides and require alternative configuration methods.

Conclusion

Programmatic endpoint redirection provides a robust solution for using VPC endpoints when private DNS is not available or feasible. This approach offers the security and cost benefits of private connectivity while maintaining flexibility across diverse network architectures.

By leveraging AWS SDK endpoint override capabilities, organizations can achieve private connectivity to AWS services without the complexity of extending private DNS across network boundaries.