The current era of generative AI is defined by a migration toward the Diffusion Transformer, or DiT. As the industry moves away from traditional U-Net architectures to embrace the scalability of Transformers, developers have encountered a persistent, ghost-like problem: visual artifacts that defy simple tuning. These glitches—stray pixels, unnatural color clumps, and structural collapses—often appear not because the model lacks data, but because the internal numerical flow of the network is being hijacked. The community has long suspected that something was happening within the latent space, but the exact mechanism of this failure remained elusive until now.
The High-Norm Token Hijack in RAE-DiT
Recent analysis into the internal behavior of Diffusion Transformers has revealed the existence of outlier tokens, specifically high-norm tokens, which act as numerical black holes within the network. In a standard RAE-DiT (Representation Autoencoder-DiT) pipeline, the process begins with a pre-trained Vision Transformer (ViT) encoder that compresses an image into a latent space. Researchers discovered that these ViT encoders frequently produce tokens with abnormally long vector lengths. While these high-norm tokens possess very little actual local semantic information, they exert a disproportionate influence over the Transformer's attention mechanism.
Because the attention mechanism relies on dot-product similarity, these extreme numerical values allow a handful of tokens to dominate the attention weights. Essentially, these outliers absorb the attention that should be distributed across the rest of the image, effectively silencing the meaningful local details. This phenomenon is not limited to the encoder. The DiT denoiser, which is responsible for iteratively removing noise to reconstruct the image, exhibits the same behavior. The distortion is most acute in the intermediate layers of the denoiser, where the process of abstracting input data causes certain token values to amplify uncontrollably. This creates a cascading failure: the encoder passes distorted representations to the denoiser, and the denoiser adds its own internal outliers to the mix, resulting in a total breakdown of the information flow.
Why Masking Fails and the DSR Solution
When faced with extreme values, the intuitive engineering response is masking—simply identifying the tokens with the highest norms and removing them from the calculation. However, experiments show that simple masking does not meaningfully improve image generation quality. The reason for this failure lies in what researchers call corrupted local patch semantics. The problem is not merely that a number is too large; it is that by the time a token becomes a high-norm outlier, the actual semantic information for that specific patch of the image has already been destroyed or distorted. Removing the token creates a hole in the data, breaking the spatial continuity of the image and introducing new instabilities into the inference process.
To solve this, researchers introduced Dual-Stage Registers (DSR), a structural intervention that redirects attention rather than deleting data. Instead of trying to erase the outliers, DSR provides the model with virtual storage spaces, or registers, that act as attention sinks. These registers are designed to absorb the excessive attention weights that would otherwise be hijacked by outlier tokens, allowing the remaining attention to be distributed naturally among the valid, information-rich tokens. This is achieved through a three-tier intervention strategy.
First, Trained Registers are integrated and optimized during the model's training phase. These act as a permanent buffer, teaching the model to maintain a balanced contextual understanding even when extreme values appear. Second, Recursive Test-time Registers are deployed during the inference stage when model weights are frozen. These registers use a recursive structure, feeding the results of previous operations back into the input to gradually normalize the token distribution without requiring any retraining. Finally, Diffusion Registers are specifically tailored for the denoiser's noise-removal process. They monitor the points where token norms shift abruptly during the diffusion steps, absorbing the spikes to prevent the pixel-level artifacts that typically plague high-resolution generations.
By applying DSR across the RAE-DiT pipeline, the system creates a double-layered defense. The encoder's registers clean up the initial latent representations, and the denoiser's registers prevent visual distortion during the final reconstruction. Unlike masking, DSR does not create information gaps; it structurally redesigns the flow of attention so the model can ignore the noise of the outliers while preserving the integrity of the local patch semantics.
Validation using the ImageNet dataset and large-scale Text-to-Image (T2I) tasks confirms that DSR significantly reduces visual artifacts such as abnormal dots, lines, and color bleeding. In complex T2I prompts, the stability of the image composition improves, and the variance in quality across different prompt complexities is narrowed. For engineers optimizing DiT models, the implementation path is clear: use Trained Registers for full-scale training, Recursive Test-time Registers for frozen-weight inference, and Diffusion Registers to eliminate pixel-level noise in the denoiser. By measuring the token norm distribution in intermediate layers, developers can now precisely diagnose outlier distortion and deploy the corresponding register to stabilize the generative process.
