Introducing Hybrid Cache for Amazon S3: An Intelligent Caching Proxy for Amazon S3

Updated 2026-05-11

Important: Hybrid Cache for Amazon S3 is sample code provided for demonstration and educational purposes. It is not intended for production use without thorough testing, security review, and validation for your specific use case and environment. You are responsible for evaluating whether this solution meets your requirements before deploying it.

If you run hybrid or on-premises workloads that read heavily from Amazon S3, you've felt the friction: repeated data transfer costs, latency on every fetch, and no good way to cache without breaking your IAM policies. Today I'm open-sourcing Hybrid Cache for Amazon S3, a transparent caching proxy that sits between your S3 clients and Amazon S3, accelerating reads while keeping your existing authentication intact.

The Problem

Organizations in healthcare, life sciences, quantitative trading, autonomous driving, and other data-intensive fields routinely pull large datasets from S3. When dozens or hundreds of clients request the same objects, every request triggers a full round-trip. That means redundant data transfer out (DTO) charges, latency that compounds under concurrency, and wasted bandwidth on data you already have on-premises.

Existing caching solutions typically require configuring the cache with its own AWS credentials, breaking the IAM-based access model that S3 provides.

How Hybrid Cache for Amazon S3 Works

Hybrid Cache for Amazon S3 is a high-performance Rust proxy that intercepts HTTP requests destined for S3 endpoints, providing a multi-tier cache (RAM and disk) while acting as a transparent forwarder. The proxy never holds AWS credentials — it forwards each client's signed request to S3, letting S3 handle authentication and authorization. When the response comes back, the proxy caches it locally so subsequent requests are served without touching S3.

Clients need minimal configuration. Set HTTP_PROXY=http://proxy:3128 to route traffic without any DNS changes, or point your DNS at the proxy for transparent interception. Either way, clients set AWS_ENDPOINT_URL_S3=http://s3.<region>.amazonaws.com (or --endpoint-url per command) so requests use HTTP and flow through the caching pipeline — HTTPS requests bypass the cache via TCP passthrough. Existing AWS CLI commands, SDKs, and tools like Mountpoint for Amazon S3 work without further modification. HTTPS traffic on port 443 passes through untouched (TCP tunneling), while HTTP requests and requests through the TLS proxy listener (port 3129) get the full benefit of caching. The TLS listener encrypts client-to-proxy traffic using the proxy's own certificate, so caching no longer requires plaintext HTTP between client and proxy. All traffic between the proxy and S3 is encrypted with HTTPS.

For private connectivity, endpoint_overrides routes outbound S3 traffic through VPC interface endpoints (PrivateLink). Suffix patterns like *.s3.us-west-2.amazonaws.com cover every virtual-hosted bucket in a region with a single entry.

What Makes It Different

There are three design choices that set Hybrid Cache for Amazon S3 apart from typical caching layers.

First, transparent authentication. The proxy reuses the client's own request signatures — no cache credentials to manage, no separate IAM roles. For environments requiring per-request authorization enforcement, set TTL to zero: every request revalidates with S3 via conditional headers, saving bandwidth on 304 Not Modified responses.

Second, shared file storage. Multiple proxy instances share cached data through any NFS-compatible volume. One instance caches an object, every other instance serves it immediately. Instances are stateless and coordinate through the shared filesystem, so you can add or remove them without reconfiguration. For high availability, proxy instances register as multi-value DNS records, mirroring S3's own DNS resolution model. The AWS Common Runtime (CRT) in AWS CLI v2 and modern SDKs automatically distributes requests across all resolved IPs and retries against alternate IPs on connection failure — no load balancer required.

Third, unified range storage. All cached data — PUTs, GETs, byte-range requests, multipart uploads — shares a common format. Upload via multipart, download as a full object or arbitrary byte ranges: the cache serves all of these without re-fetching. This enables resumable downloads — if a transfer is interrupted, the client can resume and the proxy serves already-cached ranges locally while fetching only the remainder from S3.

Performance in Practice

Testing with 100 clients concurrently downloading the same set of objects cross-region (eu-west-1 bucket, us-west-2 proxies), warm-cache performance delivered 27% higher throughput and 1.7× lower p95 latency compared to direct S3 access. During the cold-cache run, download coordination coalesced concurrent requests for the same uncached object — only one client fetched from S3 while the others waited and served from cache.

On an 8-node m6in.2xlarge proxy fleet with a shared FSx for OpenZFS volume, cross-region cache-miss throughput sustains ~2.4 GiB/s end-to-end.

Large files stream directly to clients while being cached in the background, keeping memory usage constant regardless of file size with sub-100ms first-byte latency.

Under the Hood

Hybrid Cache for Amazon S3 is written in Rust. Both RAM and disk tiers use a TinyLFU-inspired eviction algorithm for object range data, and LRU for metadata. The disk tier uses BLAKE3-based hash sharding that scales to billions of cached objects per bucket. LZ4 compression is content-aware: text and source code get compressed, while already-compressed formats like JPEG and ZIP are wrapped in frame format for integrity checksums without wasting CPU.

Connection pooling with per-IP isolation and DNS refresh keeps upstream connections efficient. A built-in dashboard provides real-time cache statistics, and metrics are exportable via Prometheus-format endpoints or OTLP to CloudWatch, Grafana, or any OpenTelemetry-compatible collector.

Cache behavior is configurable at the bucket and prefix level — TTLs, read/write caching, compression, and RAM cache eligibility can all be overridden per bucket or per key prefix via JSON settings files that hot-reload without restarting the proxy. Settings cascade from prefix to bucket to global defaults.

Getting Started

The proxy is available under the MIT-0 license (MIT No Attribution). Clone the repo, build with Cargo, and you're running in minutes:

git clone https://github.com/aws-samples/sample-s3-hybrid-cache
cd sample-s3-hybrid-cache
cargo build --release

# Write a minimal proxy_only config (no sudo, no DNS or hosts changes)
cat > ./config/quickstart.yaml <<'EOF'
server:
  mode: "proxy_only"
  proxy_port: 3128
cache:
  cache_dir: "./tmp/cache"
  max_cache_size: 524288000
logging:
  access_log_dir: "./tmp/logs/access"
  app_log_dir: "./tmp/logs/app"
EOF

cargo run --release -- -c ./config/quickstart.yaml

Then in another shell:

export HTTP_PROXY=http://127.0.0.1:3128
export AWS_ENDPOINT_URL_S3=http://s3.<region>.amazonaws.com
aws s3 cp s3://your-bucket/key ./local

proxy_only mode listens on port 3128 with no sudo and no DNS changes. TLS is unnecessary for a localhost proxy — the client-to-proxy hop is loopback. On EC2, also set NO_PROXY=169.254.169.254 so IMDS credential retrieval bypasses the proxy.

For production deployments requiring high availability, run in standard mode (ports 80/443, requires sudo) and route traffic via a DNS zone with multi-value A records — the AWS CRT client distributes requests across all proxy IPs and retries on failure, no load balancer needed. When the proxy is on a different host than the client, use HTTP_PROXY=https://proxy:3129 with the TLS proxy listener enabled for encrypted client-to-proxy traffic (a load balancer is required for HA with HTTP_PROXY).

The repo includes documentation covering architecture, caching, compression, connection pooling, deployment, and configuration. Example bucket settings files demonstrate common patterns like static asset caching, zero-TTL revalidation, and allowlist-based cache control.

Who Should Use This

Hybrid Cache for Amazon S3 is a good fit for on-premises or hybrid workloads that repeatedly read from S3, especially when multiple users or processes access the same datasets. It's useful when you need to preserve S3's native IAM authentication, leverage existing NFS storage for caching, or reduce data transfer costs.

Reminder: This is sample code. Before deploying into any production environment, thoroughly test and validate that it meets your performance, security, and reliability requirements. Review the Security Considerations section for important guidance on network security, shared cache access, and appropriate deployment scenarios.

Check out the repo at github.com/aws-samples/sample-s3-hybrid-cache and the Getting Started guide to try it out. I welcome feedback, bug reports, and feature requests via issues and pull requests.