Introducing S3 Hybrid Cache: An Intelligent Caching Proxy for Amazon S3

Important: S3 Hybrid Cache is sample code provided for demonstration and educational purposes. It is not intended for production use without thorough testing, security review, and validation for your specific use case and environment. You are responsible for evaluating whether this solution meets your requirements before deploying it.

If you run hybrid or on-premises workloads that read heavily from Amazon S3, you've felt the friction: repeated data transfer costs, latency on every fetch, and no good way to cache without breaking your IAM policies. Today we're open-sourcing S3 Hybrid Cache, a transparent caching proxy that sits between your S3 clients and Amazon S3, accelerating reads while keeping your existing authentication intact.

The Problem

Organizations in healthcare, life sciences, quantitative trading, autonomous driving, and other data-intensive fields routinely pull large datasets from S3. When dozens or hundreds of clients request the same objects, every request triggers a full round-trip. That means redundant data transfer out (DTO) charges, latency that compounds under concurrency, and wasted bandwidth on data you already have on-premises.

Existing caching solutions typically require configuring the cache with its own AWS credentials, breaking the IAM-based access model that S3 provides.

How S3 Hybrid Cache Works

S3 Hybrid Cache is a high-performance Rust proxy that intercepts HTTP requests destined for S3 endpoints, providing a multi-tier cache (RAM and disk) while acting as a transparent forwarder. The proxy never holds AWS credentials — it forwards each client's signed request to S3, letting S3 handle authentication and authorization. When the response comes back, the proxy caches it locally so subsequent requests are served without touching S3.

Clients need no changes. Point your DNS or hosts file at the proxy, and existing AWS CLI commands, SDKs, and tools like Mountpoint for Amazon S3 work without modification. HTTPS traffic passes through untouched (TCP tunneling), while requests to HTTP endpoints get the full benefit of caching. All traffic between the proxy and S3 is encrypted with HTTPS.

What Makes It Different

There are three design choices that set S3 Hybrid Cache apart from typical caching layers.

First, transparent authentication. The proxy reuses the client's own request signatures — no cache credentials to manage, no separate IAM roles. For environments requiring per-request authorization enforcement, set TTL to zero: every request revalidates with S3 via conditional headers, saving bandwidth on 304 Not Modified responses.

Second, shared file storage. Multiple proxy instances share cached data through any NFS-compatible volume. One instance caches an object, every other instance serves it immediately. Instances are stateless and coordinate through the shared filesystem, so you can add or remove them without reconfiguration. For high availability, proxy instances register as multi-value DNS records, mirroring S3's own DNS resolution model. The AWS Common Runtime (CRT) in AWS CLI v2 and modern SDKs automatically distributes requests across all resolved IPs and retries against alternate IPs on connection failure — no load balancer required.

Third, unified range storage. All cached data — PUTs, GETs, byte-range requests, multipart uploads — shares a common format. Upload via multipart, download as a full object or arbitrary byte ranges: the cache serves all of these without re-fetching. This enables resumable downloads — if a transfer is interrupted, the client can resume and the proxy serves already-cached ranges locally while fetching only the remainder from S3.

Performance in Practice

Testing with 100 clients concurrently downloading the same set of objects, warm-cache performance delivered 22% higher throughput and 2.4x lower p95 latency compared to direct S3 access. During the cold-cache run, download coordination coalesced 94% of concurrent requests for uncached objects — only 272 S3 fetches served 137 GB to clients (95% data transfer savings). Zero errors across 30,000+ requests.

Large files stream directly to clients while being cached in the background, keeping memory usage constant regardless of file size with sub-100ms first-byte latency.

Under the Hood

S3 Hybrid Cache is written in Rust. Both RAM and disk tiers use a TinyLFU-inspired eviction algorithm for object range data, and LRU for metadata. The disk tier uses BLAKE3-based hash sharding that scales to billions of cached objects per bucket. LZ4 compression is content-aware: text and source code get compressed, while already-compressed formats like JPEG and ZIP are wrapped in frame format for integrity checksums without wasting CPU.

Connection pooling with per-IP isolation and DNS refresh keeps upstream connections efficient. A built-in dashboard provides real-time cache statistics, and metrics are exportable via Prometheus-format endpoints or OTLP to CloudWatch, Grafana, or any OpenTelemetry-compatible collector.

Cache behavior is configurable at the bucket and prefix level — TTLs, read/write caching, compression, and RAM cache eligibility can all be overridden per bucket or per key prefix via JSON settings files that hot-reload without restarting the proxy. Settings cascade from prefix to bucket to global defaults.

Getting Started

The proxy is available under the MIT-0 license (MIT No Attribution). Clone the repo, build with Cargo, and you're running in minutes:

git clone https://github.com/aws-samples/sample-s3-hybrid-cache
cd sample-s3-hybrid-cache
cargo build --release
sudo cargo run --release -- -c config/config.example.yaml

Point your S3 traffic at the proxy by adding a hosts file entry (for single proxies and test) or configuring a DNS zone, set AWS_ENDPOINT_URL_S3=http://s3.<region>.amazonaws.com, and your existing AWS CLI commands start caching automatically.

The repo includes documentation covering architecture, caching, compression, connection pooling, deployment, and configuration. Example bucket settings files demonstrate common patterns like static asset caching, zero-TTL revalidation, and allowlist-based cache control.

Who Should Use This

S3 Hybrid Cache is a good fit for on-premises or hybrid workloads that repeatedly read from S3, especially when multiple users or processes access the same datasets. It's useful when you need to preserve S3's native IAM authentication, leverage existing NFS storage for caching, or reduce data transfer costs.

Reminder: This is sample code. Before deploying into any production environment, thoroughly test and validate that it meets your performance, security, and reliability requirements. Review the Security Considerations section for important guidance on network security, shared cache access, and appropriate deployment scenarios.

Check out the repo at github.com/aws-samples/sample-s3-hybrid-cache and the Getting Started guide to try it out. We welcome feedback, bug reports, and feature requests via issues and pull requests.