The modern LLM deployment cycle is often a brutal trade-off between latency and the monthly cloud bill. For engineering teams building RAG pipelines or complex multi-turn agents, the 'GPU memory tax' is a constant pressure. When a model's KV cache exceeds the available HBM, the system is forced to re-calculate the prompt from scratch for every single request—a process that spikes the Time-To-First-Token (TTFT) and degrades the user experience. Until now, the industry's primary solution was brute force: migrating to the most expensive, high-memory GPU instances available just to keep the cache resident.

The Architecture of a Three-Tier Memory Hierarchy

To break this dependency on ultra-high-end hardware, SageMaker HyperPod has implemented a sophisticated tiered storage strategy that offloads the KV cache across three distinct layers: L0, L1, and L2. This hierarchy is designed to maximize the utility of every byte of memory, from the fastest HBM to the most abundant NVMe storage.

At the top is L0, the vLLM Paged-Attention layer residing directly in GPU memory. This provides the lowest possible latency but is severely limited by physical capacity. For example, running a 7B model in bf16 on a 48GB GPU consumes roughly 14GB for weights, leaving ample room. However, a 32B model requires approximately 64GB for weights alone, meaning it cannot fit on a single GPU. Even after sharding, the remaining space for the KV cache is minimal, leading to 'L0 pressure' where cache blocks are evicted almost as soon as they are created.

When L0 is exhausted, blocks move to L1, a temporary storage layer in the host's DRAM managed by LMCache. This process is handled automatically by the SageMaker HyperPod Inference Operator. Developers can activate this by setting `enableL1Cache: true` within the `InferenceEndpointConfig` CRD. To balance system stability, the `InstanceMemoryAllocationPercentage` determines how much DRAM is dedicated to L1, with a recommended starting value of 20%.

Finally, the L2 layer serves as the persistent, shared backbone. By leveraging Curvine, HyperPod pools the local NVMe drives of multiple nodes into a single, unified namespace. This transforms volatile, pod-specific memory into a distributed pool that functions with speeds approaching local disk access. In practical tests with prompts of approximately 1,900 tokens, the cross-node L2 read latency was measured at roughly 56ms. This tiered approach allows the system to achieve a TTFT improvement of up to 2.7x by drastically reducing the need for expensive re-prefill operations.

Breaking the Isolation of vLLM Replicas

The true technical pivot here is the shift from isolated pod memory to a shared state. In a standard horizontally scaled vLLM deployment, each replica maintains its own independent GPU blocks and CPU spill areas. If a request is routed to Replica A, and the subsequent turn is routed to Replica B, Replica B has no access to the previous KV cache. This results in a 'cold start' for every request that hits a different pod, forcing the system to re-calculate the same system prompts and context windows repeatedly.

By mounting the Curvine distributed file system as a ReadWriteMany PVC across all inference pods, SageMaker HyperPod eliminates this isolation. Curvine uses a FUSE (Filesystem in Userspace) client to present the pooled NVMe drives as a standard directory. LMCache then accesses this pool via the `fs://` connector. Because every pod shares the same namespace, a KV block written by one replica is immediately available to all others. This architecture achieves a 100% cross-pod cache hit rate, effectively turning a cluster of independent pods into a single, massive cache entity.

Curvine operates via a Primary Node, which manages metadata and journaling on Amazon EBS, and Worker nodes, which store the actual data on local NVMe drives. The data is stored at the following path:

/opt/dlami/nvme/curvine-data

Even if a worker node fails and a portion of the cache is lost, the system remains resilient; since KV blocks are reproducible from the input prompt, the worst-case scenario is a temporary return to re-prefill latency rather than a service outage.

To ensure the right request hits the right cache, the system employs intelligent routing. It supports Prefix-aware strategies, which use a prompt prefix tree to route requests to nodes likely to hold the relevant cache, and KV-aware strategies, which query the actual cache state of the workers. Because this happens at the infrastructure layer, no changes to the client-side API or application code are required.

Operational Implementation and the 40% Threshold

Integrating this tiered system into an existing pipeline requires a specific configuration patch. While the `InferenceEndpointConfig` CRD natively supports `redis` or `tieredstorage` for the `l2CacheBackend` field, utilizing the Curvine FUSE mount requires modifying the environment variables of the vLLM container. Specifically, the `LMCACHE_REMOTE_URL` must be pointed to the Curvine mount path:

LMCACHE_REMOTE_URL=fs://localhost:0/mnt/curvine/l2cache/

The entire lifecycle is managed by the Inference Operator installed as an Amazon EKS add-on. This operator orchestrates the deployment of the vLLM pods, the LMCache sidecars, and the router. The sidecar container handles the heavy lifting of moving data between L1 and L2, ensuring the main inference engine remains agnostic to the physical location of the cache.

For teams deciding whether to adopt this architecture, the primary metric is the prompt overlap rate—the percentage of shared leading tokens across requests. In environments where prompt overlap exceeds 40%, the reduction in TTFT becomes significant. This is typical for RAG applications where a large, static document set is shared across many queries, or for chatbots with extensive system prompts.

When this 40% threshold is met, the economic argument becomes undeniable. Instead of paying the premium for P5 instances to accommodate massive HBM requirements, teams can deploy their workloads on the more cost-effective G6e instances. By augmenting limited GPU memory with a shared NVMe L2 cache, organizations can maintain high performance while slashing the operational cost per endpoint.

This shift represents a broader trend in AI infrastructure: moving away from the pursuit of the 'single biggest GPU' and toward intelligent, tiered memory orchestration that treats the entire cluster as a unified compute and storage resource.