The current race to scale large language models is no longer just about parameter counts or training data quality; it is a war over VRAM. Engineers building long-context agents and sophisticated RAG systems are hitting a physical wall where the cost of adding more H100s just to accommodate a larger context window becomes economically unsustainable. Every single token of context consumes precious GPU memory, and when that memory runs out, the system either crashes or falls back to agonizingly slow re-computation. This bottleneck has turned the GPU memory wall into the primary obstacle for production-grade AI agents that need to remember thousands of lines of code or hours of conversation history.
The Architecture of Augmented Memory
Weka is attempting to break this bottleneck with the release of NeuralMesh 6, a software platform paired with a dedicated hardware line called Wekapod 3. At the center of this release is a concept called the Augmented Memory Grid (AMG). Rather than relying solely on the expensive High Bandwidth Memory (HBM) found on GPUs, AMG aggregates relatively inexpensive NAND flash storage and presents it to the system as an extension of GPU memory. This allows the infrastructure to handle massive datasets without requiring a linear increase in GPU hardware.
To support the demands of modern AI clouds, NeuralMesh 6 introduces a dual-layer isolation strategy for multi-tenancy. The first layer consists of Composable Clusters, which provide hardware-level isolation to ensure that one tenant's workload cannot interfere with another's physical resources. The second layer is Virtual Multi-tenancy, which operates at the network level via Weka's RDMA (Remote Direct Memory Access) fabric. This architecture allows a single cluster to support over 1,000 tenants, with provisioning times clocked at under 30 minutes. When scaled across a larger environment containing 50 Composable Clusters, a single unified cluster can support up to 50,000 tenants.
Efficiency extends to the data path itself. Traditionally, AI pipelines operate two separate silos: a file storage path for training and fine-tuning, and an object storage path (typically S3) for inference and cloud-native tools. Moving data between these two requires a gateway that often results in redundant data copies and increased latency. NeuralMesh 6 eliminates this conversion layer by unifying file and object storage into a single physical data layer. This allows the system to read the same data directly through either path. Targeting non-AWS GPU clouds such as Lambda, Nebius, G42, and CoreWeave, Weka claims this unified approach delivers performance approximately 100 times higher than traditional S3 setups, while shifting the pricing model from API call counts to a more predictable capacity-based structure.
Solving the Redundancy Crisis in Inference
To understand why offloading to NAND flash is a breakthrough, one must look at the mechanics of LLM inference. The process is split into two distinct phases: the Prefill phase, where the model calculates the attention for the input tokens, and the Decode phase, where it generates the output tokens one by one. The Prefill phase is computationally expensive, requiring massive matrix multiplications. The Decode phase is relatively light, but it relies on the results of the Prefill phase, which are stored as the KV (Key-Value) cache.
In multi-turn conversations or complex coding tasks, the model must maintain the context of previous exchanges. If the KV cache is not preserved, the model must re-run the Prefill phase for the entire conversation history every time a new prompt is entered. The computational waste is exponential. Weka's analysis shows that in a 10-turn conversation without caching, the system performs up to 100 redundant calculations. By the 20th turn, that number jumps to 400. This is why long-context windows often lead to a visible spike in latency and a surge in compute costs.
NeuralMesh 6 solves this by storing 100% of these pre-calculated KV tokens in NAND flash via the Augmented Memory Grid. By utilizing a volume of flash storage that far exceeds what could be affordably allocated as shared GPU memory, the model can instantly retrieve previous context from storage rather than recalculating it. To balance the trade-off between speed and cost, Weka employs AlloyFlash. This system mixes TLC (Triple-Level Cell) flash for latency-sensitive operations and QLC (Quad-Level Cell) flash for high-capacity workloads within a single cluster. The system automatically routes data based on the priority of the task, ensuring that the cost per terabyte remains low without sacrificing the responsiveness of the AI.
This optimization is further supported by Always-On data reduction, which is integrated as a default setting to maximize storage efficiency. The impact extends beyond just inference speed to the very way infrastructure is deployed. Weka has implemented a metadata-first replication strategy. In traditional setups, moving massive datasets to a new GPU environment could take days or even a month. With metadata-first replication, the system replicates the file structure first, allowing users to browse the data immediately. The actual data is then hydrated—pulled from the source—only when it is specifically accessed. This shift allows a service to go live within one hour of a new GPU allocation.
For enterprises building internal Copilots, customer service agents, or RAG-based software engineering assistants, this removes the need to artificially limit context windows to save on VRAM. Instead of investing in more GPUs to solve a memory problem, organizations can scale their context capabilities through the storage layer, effectively decoupling the intelligence of the model from the physical limits of the GPU's onboard memory.
This shift transforms the GPU from a primary storage vessel for context into a pure compute engine, signaling a move toward a more modular and cost-effective AI infrastructure.
