Enterprise search engineers are currently hitting a wall with multimodal RAG. The industry has shifted toward late-interaction models that treat document pages as images to preserve layout and visual context, but this progress comes with a punishing tax. Indexing millions of pages requires massive GPU clusters, and the resulting vector databases often swell to unmanageable sizes, making high-precision document retrieval a luxury reserved for those with nearly infinite infrastructure budgets. The tension lies in the trade-off between the rich visual understanding of large vision-language models and the operational reality of storage and latency.

The Architecture of Efficiency and the End of OCR

NeoMME enters this space by stripping away the traditional overhead of vision-language models. Unlike most contemporary retrievers that rely on a heavy vision tower paired with a causal language model, NeoMME utilizes a single bidirectional transformer to process text tokens and image patches within a unified computational path. This architectural consolidation allows the model to bypass the need for separate vision and language components, drastically reducing the operational overhead during pre-training, fine-tuning, and serving. The result is a streamlined system available in two scales: a 260M parameter version and an 800M parameter version.

In practical deployment, the performance gains are immediate. On an NVIDIA L40S GPU, the NeoMME-Retriever-260M model encodes 2048×2048 resolution images at a rate of approximately 51 pages per second. This represents a 2x increase in throughput compared to the 26 pages per second recorded by ColModernVBERT. For an organization indexing tens of thousands of corporate documents, this speedup effectively halves the total indexing window and lowers the associated infrastructure costs.

One of the most significant shifts in the NeoMME pipeline is the total removal of Optical Character Recognition (OCR) pre-processing. Instead of attempting to extract text from a PDF—a process that often destroys the structural integrity of tables, charts, and font-based hierarchies—NeoMME treats the entire page as a single image. By preserving the visual layout, the model maintains the context of professional technical documents and financial reports where the spatial relationship between data points is as important as the text itself. The model is released under the Apache 2.0 license, with all checkpoints distributed via the Hugging Face Transformers library for immediate integration.

To achieve this, the team abandoned the common practice of transfer learning from pre-trained models. Instead, NeoMME was trained from scratch using a masked discrete-diffusion objective function. For text-only examples, the model samples a corruption rate between 0 and 1 to independently mask eligible tokens. Through a denoising process, the model learns to predict the original values of these masked tokens, forcing it to learn the intrinsic structure of both text and image patches in a shared embedding space. The training set was massive, consisting of 524 billion packed input tokens, including 290 billion text-only tokens. This mixture of multilingual text, code, mathematical expressions, and document images ensures the model understands both linguistic logic and visual layout. To maintain stability across this vast dataset, the team employed the NorMuon optimizer, ensuring faster convergence and higher training efficiency.

Breaking the Pareto Frontier of Retrieval

When analyzing the benchmarks, the disparity between model size and performance becomes the central story. On the ViDoRe v3 benchmark, the NeoMME-Retriever-260M achieved an nDCG@10 score of 0.523. To put this in perspective, the much larger ColQwen2.5 scored only 0.002 higher, yet it possesses roughly 14 times more parameters. NeoMME has effectively compressed the intelligence of a giant model into a fraction of the size without sacrificing retrieval accuracy.

The 800M version pushes these boundaries further, reaching an nDCG@10 of 0.556. This outperforms ColPali v1.3 despite having 3.6 times fewer parameters and remains competitive with the Vultron Retriever Flash (0.8B), which scored 0.565. Even on older benchmarks like ViDoRe v1 and v2, the 260M model consistently outperformed ColModernVBERT and the larger ColSmol-500M. This suggests that for document retrieval, simply adding parameters yields diminishing returns; the architectural choice of a bidirectional transformer and the specific diffusion-based training are more critical than raw scale.

However, the most disruptive innovation is how NeoMME handles the storage crisis inherent in late-interaction embeddings. In a standard setup, a single 2048×2048 image generates roughly 4,200 vectors. Using float32 precision, this requires about 2.1MB per page, though ViDoRe v3 measurements show an average of 1.5MB. For a million-page corpus, this creates a storage nightmare. NeoMME solves this through a combination of hierarchical token pooling and asymmetric quantization.

By implementing a pooling factor of 10 and int8 quantization (referred to as Setting A), the model reduces the per-page storage from 1.5MB to 39kB—a 39x reduction—while maintaining over 99% of the original nDCG@10 performance. For environments with extreme constraints, Setting B applies a pooling factor of 8 and stores documents in binary format while keeping queries in int8. This collapses the storage requirement to just 6kB per page. This is a 255x reduction in index size, yet it still retains 95% of the retrieval accuracy. Reducing a page to 6kB transforms the operational profile of the system, allowing entire indices to reside in memory or drastically reducing disk I/O overhead.

This capability allows developers to build high-performance search engines in hardware-constrained environments where previously only simple dense retrievers were viable. By moving the model to the Pareto frontier—the optimal balance between size and performance—NeoMME makes vision-based RAG accessible for production-scale deployments.

To maximize this efficiency in the real world, NeoMME is designed to return both dense embeddings and late-interaction embeddings in a single forward pass. Dense embeddings are lightweight and ideal for rapid initial filtering, while late-interaction embeddings provide the high-precision reranking needed for complex queries. The recommended production workflow is a two-stage pipeline: use dense embeddings within an Approximate Nearest Neighbor (ANN) index to extract a small set of candidates from millions of documents, then apply late-interaction embeddings to rerank only those few candidates. This hybrid approach, supported by libraries like NextPlaid and Sentence Transformers v6, ensures that the system maintains maximum precision without the computational cost of calculating late interactions for every document in the database.

This shift toward ultra-compressed, high-throughput multimodal encoders signals the end of the era where OCR was a mandatory bottleneck in document AI.