For years, the boundary between consumer hardware and frontier-scale artificial intelligence has been defined by a brutal physical limit: VRAM. Developers accustomed to running 7B or 70B parameter models on local workstations have long accepted that trillion-parameter models are the exclusive domain of massive H100 clusters and cloud providers. The moment a model's weight file exceeds the available memory, the system typically crashes with an out-of-memory error or slows to a glacial pace as the operating system desperately swaps memory to disk. This week, however, a new architectural approach suggests that the bottleneck isn't the amount of RAM, but how the engine interacts with the storage layer.

The Architecture of Weight-Aware Streaming

WASTE, the Weight-Aware Streaming Tensor Engine, is designed to decouple model size from memory capacity. Built using the C11 compiler and make, the engine is an embeddable inference system that strips away almost all external dependencies, relying solely on `libc` and `pthreads`. This lean implementation allows it to support the Kimi K3 open-weight model, a behemoth featuring 2.78 trillion parameters. To make this manageable, WASTE converts the original 1.42TB of shard files into a single, optimized container measuring 982GiB. Crucially, this is achieved without using distillation or pruning; the engine maintains the full integrity of the model weights while enabling execution on consumer-grade hardware.

Memory requirements for WASTE are scaled based on the desired context length. For a 4K context, the engine requires a minimum of 29.05GB of RAM. As the context window expands to 32K, the requirement rises to 30.54GB, and for 128K, it hits 35.63GB. At the extreme end, a 1M context window demands 83.21GB of RAM. The resident trunk weights alone occupy 27.28GB, and the model typically loads in approximately 20 seconds. For those with more modest hardware, the Kimi-Linear 48B model can be run using a 19GiB container and a minimum of 1.87GB of RAM, delivering a throughput of 10.7 tok/s.

The engine exposes a streamlined API consisting of 26 functions that handle model initialization, RAM limit configuration, token generation, and session management. By implementing the entire inference path in C and completely removing Python from the runtime, WASTE eliminates the overhead typically associated with high-level wrappers. PyTorch is utilized only during the initial model conversion phase. The project is released under the Apache 2.0 license, making it accessible for wide integration into local AI pipelines.

Bypassing the OS Cache for NVMe Streaming

The core innovation of WASTE lies in its exploitation of the Mixture of Experts (MoE) architecture. In an MoE model, only a small fraction of the total parameters—roughly 4% per token in the case of Kimi K3—are active at any given time. WASTE leverages this by treating the NVMe SSD as an extension of memory, streaming only the necessary active weights into RAM in real-time rather than attempting to load the entire model. The `.waste` container is structured with a JSON manifest, resident trunk weights, and layer-specific expert banks. Each expert record is 4KiB aligned, allowing the engine to fetch the gate, up, and down matrices using a single `pread` operation.

To prevent the operating system from interfering with this precision streaming, WASTE bypasses the standard page cache. It employs `F_NOCACHE` on macOS, `O_DIRECT` on Linux, and `FILE_FLAG_NO_BUFFERING` on Windows. This ensures that the RAM usage remains predictable and prevents the OS from attempting to cache massive chunks of the model, which would otherwise distort hit rates and trigger system instability. To further optimize throughput, the engine implements router lookahead and computation overlapping. When a router in one layer determines the required expert IDs, WASTE requests the data via separate threads, feeding the weights into the computation pipeline the moment they arrive. This strategy provides a 1.6x performance boost for Kimi K3.

One of the most significant breakthroughs is the predictive loading of experts. By executing the router for the subsequent layer using the current hidden state, WASTE can pre-load six experts before they are officially called. This mechanism increases the expert cache hit rate from 14% to 38% without sacrificing accuracy, as the actual router still makes the final determination for the output logits. To keep the footprint small, the engine uses 3-stage Residual Vector Quantization (RVQ), allocating 3.00 bits per weight. Instead of reconstructing the full matrix, WASTE uses partial inner product tables. The trunk area is kept at 4-bit and 8-bit precision to prevent output collapse, resulting in a logit difference of only 3.6e-06 compared to a full PyTorch implementation.

Memory efficiency is further pushed by the combination of Kimi Delta Attention (KDA) and gated multi-head latent attention (MLA) in a 3:1 ratio. This optimization reduces the KV cache size for a 4K context from 11.25GB down to 0.21GB, a reduction of approximately 53 times. However, this efficiency comes with a strict hardware dependency. Benchmarks on a 64GB MacBook Pro (M5 Pro) show that setting a RAM budget of 46GB yields 0.45 to 0.62 tok/s. If the budget is increased beyond 52GB, performance collapses to 0.02 to 0.15 tok/s. This is not a cache miss issue, but a result of OS paging, where the system begins swapping memory to disk, creating a massive bottleneck. Furthermore, using an external USB enclosure (0.94GB/s) instead of an internal NVMe SSD (12.78GB/s) increases latency to roughly 13 seconds per token due to the 17GB of read operations required per token.

Beyond text, WASTE supports multimodal processing via a 401M parameter Vision Transformer (ViT). Encoding a 1024-patch image takes approximately 15.7 seconds, as image embeddings must pass through the same 92 MoE layers as text prefill. For deployment, the project includes an OpenAI-compatible HTTP server that calls the C API via `ctypes`. This server supports standard endpoints like `/v1/chat/completions`, enabling streaming, tool definitions, JSON response schemas, and `thinking_effort` configurations.

The engine is verified for macOS arm64, Linux arm64, and Linux x86_64, with basic functionality confirmed for Windows x86_64 via MinGW-w64 cross-compilation. Interestingly, the Metal backend is disabled by default because the frequent, small matrix operations inherent in this streaming architecture make it 22% slower than the CPU. For those converting models, the process takes about 4.7 hours on an M5 Pro using three processes, with a layer-based resume feature to handle interruptions.

For practitioners implementing WASTE, the critical variables are the RAM budget and the physical location of the model container. To avoid the performance cliff caused by OS paging, the RAM budget should be set to approximately 7/8 of the total physical RAM. Most importantly, the model container must reside on an internal NVMe SSD; any slower storage medium renders the trillion-parameter inference impractical for real-time use.