For years, the ambition of running truly large language models on a smartphone has been throttled by a single, stubborn physical constraint: the RAM wall. While mobile chips have grown exponentially more powerful, the memory overhead required to load a 30-billion-parameter model typically exceeds the total available memory of even the most premium handsets. Developers have been forced into a compromise, opting for tiny, distilled models that lack the reasoning depth of their larger counterparts, or relying on cloud APIs that sacrifice privacy and latency. This week, that paradigm shifted as a new runtime proved that the hardware barrier is no longer an absolute ceiling.

The Architecture of Swiftlet and the Qwen Integration

The breakthrough centers on Swiftlet, a specialized runtime built on Swift and Metal designed specifically to optimize the execution of the Qwen3-Next and Qwen3.5/3.6 Mixture-of-Experts (MoE) hybrid model families. By leveraging the tight integration between Apple's software stack and its silicon, Swiftlet allows massive models to operate natively on consumer hardware. The performance metrics are stark. On an iPhone 17, a 35B parameter model can now run using only 2.5GB of RAM, achieving an inference speed of approximately 1 token per second. The efficiency extends to the Mac ecosystem as well, where an 80B model can be executed with a RAM footprint of just 4.3GB.

To implement this environment, users can utilize Priv AI, an on-device AI application available on the App Store. By navigating to the Experimental Models menu, users can download the weights and establish a completely local chat environment that bypasses external servers. For those looking to build their own implementation, the project is open source, utilizing the `leonickson1/localLLM` source code to allow developers to compile and execute these models independently.

However, this accessibility comes with specific system requirements. The runtime demands Apple Silicon hardware running either macOS 14+ or iOS 17+. While the RAM usage is remarkably low, the storage requirement remains a factor because the model weights must reside on the device. A 35B model requires 18GB of available SSD space, while the 80B model requires 42GB. These storage allocations serve as the necessary container for the weight data that the runtime streams during active inference.

Weight Streaming and the Linear Attention Pivot

The ability to run a 35B model in 2.5GB of RAM is not achieved through traditional quantization alone, but through a fundamental shift in how weights are handled. Swiftlet employs a weight streaming mechanism that treats the device's RAM not as a permanent warehouse for the model, but as a high-speed cache. In a standard MoE architecture, only a fraction of the total parameters are active for any given token. Swiftlet exploits this by keeping only the small dense core—the essential computational part shared by all parameters—resident in the memory. The remaining MoE weights are streamed on-demand from the SSD storage only when the router determines they are needed for a specific calculation.

This creates a distinct functional divergence in the model's behavior. Because the generative capabilities are driven by the active parameters, the model maintains the sophisticated linguistic fluency and reasoning of a large-scale model. However, because the full breadth of the model's knowledge is not instantly available in RAM, its factual recall—the ability to retrieve specific, niche pieces of information—behaves more like that of a much smaller model. This is the inherent trade-off of weight streaming: you gain the intelligence of a giant but the memory footprint of a dwarf, at the cost of some retrieval precision and a slower 1 tok/s output speed.

To prevent the system from crashing as conversations grow longer, Swiftlet addresses the KV cache bottleneck through Gated DeltaNet. In traditional transformer models, the memory required for the KV cache grows linearly with the context length, eventually consuming all available RAM regardless of the model size. Swiftlet applies Gated DeltaNet—a linear attention mechanism—to 75% of the model's layers. By utilizing a fixed-size recurrent state, Gated DeltaNet ensures that memory consumption remains constant even as the input sequence expands. This architectural choice eliminates the memory spikes typically associated with long-context window processing, providing a stable environment for sustained on-device interaction.

This shift from static memory loading to dynamic weight streaming and linear attention transforms the smartphone from a mere terminal into a legitimate compute node for large-scale AI.