In the race to build high-performance generative AI, the focus often shifts from the model architecture to the invisible infrastructure beneath it. For engineers working on the PRX 7B model, the challenge was not just about achieving aesthetic perfection in training images, but about managing the sheer scale and diversity of data required to teach a model visual concepts, lighting, and composition at a massive scale.

Data Strategy and Quality Control

The PRX 7B team prioritized breadth over individual image perfection during the pre-training phase. By avoiding overly aggressive aesthetic filtering, the team ensured the model retained a wide conceptual range, leaving fine-grained aesthetic adjustments for the later fine-tuning and preference alignment stages. Quality control was instead managed through high-fidelity captioning using a Vision Language Model (VLM). By generating detailed descriptions, the team transformed elements like logos and text—often considered noise—into controllable attributes. This approach allowed the team to keep filtering light, removing only data that was entirely unusable.

The Role of Lance and MDS

Handling billions of image rows requires a delicate balance between searchability and high-speed streaming. The PRX team solved this by splitting their pipeline into two distinct formats: Lance for data curation and Mosaic Data Shards (MDS) for training. Lance acts as a columnar format that enables rapid vector searches and interactive indexing, allowing engineers to explore data distributions before committing to a build. Once the dataset is finalized, it is converted to MDS, which is optimized for streaming from object storage like S3 or GCS during distributed training.

This separation provides critical flexibility. While MDS is rigid—requiring a full rewrite of the dataset to add new features—Lance supports conditional pushdown and scalar indexing, making it ideal for feature engineering. The team noted that the performance of Lance depends heavily on fragmentation management; keeping the number of fragments optimized is essential to prevent query speeds from degrading as metadata operations scale.

Real-Time Encoding with Qwen3-VL

The team made a strategic shift by replacing T5Gemma with Qwen3-VL as the text encoder. Rather than pre-computing text latent representations and storing them in MDS, the team moved to real-time calculation within the training loop. While this introduced a 3% to 4% loss in throughput—adding roughly one day to a 30-day training cycle—the benefits were significant. By eliminating the pre-computation step, the team drastically reduced the storage footprint of their MDS shards, allowing the entire dataset to reside on SSD-based shared storage within their SLURM cluster. This change also removed the need to rewrite terabytes of data whenever the encoder model was updated.

Storage Efficiency and Practical Trade-offs

To further optimize infrastructure costs, the team adopted a JPEG 92 encoding strategy for all training images. Compared to lossless PNG formats, this reduced storage requirements by 3x to 10x. Internal experiments confirmed that for a 7B model, the visual quality difference between PNG and JPEG 92 was statistically insignificant and imperceptible to the human eye. While specialized models—such as those requiring high-precision shadow boundaries—still utilize PNG to maintain data purity, the team found that JPEG 92 is more than sufficient for learning general visual concepts.

By decoupling data curation from training and prioritizing flexible, format-specific workflows, the PRX team has established a scalable blueprint for large-scale model development.