For years, the prevailing wisdom in large language model deployment has been a simple, painful trade-off: you can have a model that is fast and lean, or you can have a model that is intelligent and precise. Quantization, the process of reducing the numerical precision of model weights to save memory, has traditionally been viewed as a necessary evil. Developers accept a certain percentage of performance degradation—the quantization tax—as the price for fitting a massive model onto available GPU hardware. The goal was always damage control, attempting to keep the 4-bit version as close to the 16-bit original as possible without letting the model collapse into incoherence.

The 4-Bit Model That Outran Its Original

Recent benchmarks involving the GPT-OSS 120B model have fundamentally challenged this assumption. By applying structural compression to reduce the model to 60B parameters and then quantizing it to MXFP4 precision, researchers implemented a technique called QAH. The results were unexpected: the resulting 4-bit model outperformed the original bfloat16 model in seven out of nine key benchmarks. This is not a marginal gain or a rounding error; it is a reversal of the standard efficiency curve where a smaller, lower-precision model actually becomes more accurate than the high-precision checkpoint from which it was derived.

The performance gains were most pronounced in areas where structural compression typically causes the most damage. In the AA-LCR benchmark, which measures long-context reasoning, the QAH model saw a score increase of 7.4 points. Mathematical capabilities also spiked, with the AIME 2025 benchmark showing a 5.6 point improvement. When compared against the 60B bfloat16 checkpoint, the QAH model achieved 66.5 on LiveCodeBench, effectively surpassing the original 120B model's score of 66.0. Even in the GPQA Diamond benchmark, the QAH model hit 67.4, trailing the original 120B model's 69.0 by a slim 1.6 point margin. While MMLU-Pro and SciCode remained slightly lower than the original, the gap stayed under 1.5 points. Essentially, QAH proved that a model can halve its parameter count and slash its precision to 4-bit while still punching through the performance ceiling of the full-sized original.

Breaking the Ceiling via Direct Distillation

To understand why QAH succeeds where other methods fail, one must look at the architectural bottleneck of Quantization-Aware Distillation (QAD). In a standard QAD pipeline, a model is structurally compressed and then recovered into a bfloat16 checkpoint. This recovered model then serves as the teacher for the student model. The problem is that the recovered checkpoint is already a degraded version of the original; it carries the scars of compression. Consequently, the student model is trapped by a performance ceiling defined by a flawed teacher.

QAH eliminates this middleman. Instead of learning from a recovered checkpoint, the student model distills knowledge directly from the original, full-size, high-precision teacher model. The teacher remains the untouched 120B bfloat16 model, while the student is the 60B MXFP4 model. Because these two models have different layer counts and head configurations, they are architecturally mismatched. QAH solves this by focusing on the logits—the raw output distributions—rather than the internal weights. By using KL-divergence loss to align the probability distributions of the student's outputs with those of the teacher, the student can absorb the original model's intelligence regardless of its own smaller size or lower precision.

This shift redefines quantization. It is no longer a post-processing step of numerical conversion, but a secondary distillation process that actively recovers and enhances information. To make this viable for modern LLM requirements, the researchers addressed the memory wall associated with long-context windows. Processing a 32k token healing corpus typically requires a massive memory grid for KL-divergence calculations, which often triggers out-of-memory (OOM) errors on standard GPUs. QAH implements a chunked KL-divergence loss, breaking the sequence into smaller slices and calculating the loss sequentially. This allows the model to maintain a 32k context window within a fixed GPU memory budget, ensuring that the student model captures the complex, long-range dependencies of the original teacher.

Stability and Convergence: QAH vs QAT

When compared to Quantization-Aware Training (QAT), the operational advantages of QAH become clear. In tests using the GPT-OSS 9B model quantized to MXFP4, QAH reached its peak performance of 54.9 in approximately 100 steps. In contrast, QAT required 700 steps to reach a similar peak of 54.6. QAH converges roughly seven times faster because it is not trying to find the answer from scratch; it is simply mirroring a pre-existing, high-quality distribution provided by the teacher.

More critical than speed is the issue of stability. QAT exhibits a volatile training trajectory. Once it hits its peak performance, continued training often leads to a catastrophic collapse. In observed cases, QAT performance plummeted by approximately 19 points by the 1,200-step mark. This happens because QAT relies on cross-entropy loss, which relentlessly pushes the model toward hard labels, often leading to overfitting or weight divergence in low-precision environments. QAH, however, remains remarkably stable. Even after passing its peak, the performance variance stays within 2 points. Because the student is tethered to the fixed distribution of the teacher via KL-divergence, it cannot drift into the instability that plagues QAT.

For engineers and practitioners, this removes the high-stakes gamble of early stopping. With QAT, developers must constantly monitor validation sets and stop training at the exact moment of peak performance to avoid collapse. QAH removes this operational burden, allowing for a more robust and predictable deployment pipeline. Furthermore, QAT is computationally expensive, requiring the re-execution of SFT, RLHF, and agent tuning in a low-precision environment. QAH allows the teacher's logits to be pre-calculated offline, drastically reducing the active compute resources required during the distillation phase.

By decoupling the student's learning process from the degraded recovery checkpoints and leveraging the stability of direct distribution matching, QAH transforms quantization from a lossy compression task into a performance enhancement strategy.