The AI community has a peculiar habit of turning obscure prompts into high-stakes litmus tests for intelligence. This week, the focus shifted to a specific, surreal challenge: generating a Scalable Vector Graphics (SVG) image of a pelican riding a bicycle. For developers and researchers, this is not about the art; it is about detecting a hidden pathology in modern LLMs known as benchmaxxing. As official benchmarks become increasingly contaminated by training data, the community has pivoted toward these unpredictable, community-driven prompts to see if models are actually reasoning or simply recalling a memorized answer from a viral thread.

The Anatomy of the SVG Experiment

To determine if frontier models are secretly optimizing for these viral prompts, a rigorous experiment was conducted across seven of the industry's leading models: GPT-5.6 Terra, Claude Sonnet 5, Gemini 3.5 Flash, Grok 4.5, Qwen3.7-Max, GLM-5.2, and DeepSeek V4 Pro. The researchers avoided a simple one-off test, instead implementing a grid-based methodology to isolate variables. They designed 48 distinct prompts by crossing eight different animals with six different types of vehicles. Each model generated three images per combination, resulting in a total dataset of 1,008 SVG images, which were then scored by an LLM-based judge.

The data revealed that the pelican-bicycle combination was far from a sweet spot for these models. Among the eight animals tested, the pelican ranked 6th in overall quality. The bicycle performed even worse, landing in the bottom two of all vehicle types. When combined, the pelican-bicycle prompt produced the 42nd best result out of the 48 possible combinations. To ensure these results were not coincidental, the researchers applied fixed-effects regression analysis to see if any specific model showed a statistically significant performance boost only when dealing with pelicans or bicycles. Aside from a minor, transient spike in bicycle performance from Gemini 3.5 Flash, no evidence of targeted optimization was found.

Further scrutiny focused on whether the models were relying on a memorized visual template. A common critique of the pelican-bicycle output is that the images almost always face right. However, the analysis showed that 60% of all generated images across all animal-vehicle combinations faced right, suggesting a general model bias rather than a specific memorized prompt. Other combinations actually showed higher rates of repetitive elements; for instance, the flamingo-boat combination frequently included a sun, and the otter-airplane combination often added a scarf. The pelican-bicycle outputs were, by comparison, unremarkable in their lack of consistency.

The War Against Benchmaxxing

This experiment addresses a growing tension in the AI arms race: the rise of benchmaxxing. In a market where billions of dollars in investment and user adoption hinge on a few percentage points of a benchmark score, developers face a powerful incentive to optimize for the test rather than the task. Benchmaxxing occurs when a model is trained on the specific questions and answers found in popular evaluation datasets, creating an illusion of capability. This has led to a crisis of trust in official metrics, pushing the community toward unofficial benchmarks, such as the pelican prompt popularized by Simon Willison.

The shift toward community benchmarks represents a move toward adversarial evaluation. Users are no longer satisfied with a high MMLU score; they want to see how a model handles a request it could not have possibly seen in a curated training set. The findings of this SVG experiment suggest that, at least in this instance, frontier model labs are not yet engaging in the granular task of overfitting to viral community prompts. While they may not be cheating on the pelican test, they are still pursuing a broader strategy of SVGmaxxing—improving the general ability to generate code-based imagery, a trend particularly visible in the efforts of Google and DeepMind.

This distinction is critical. There is a fundamental difference between a model that has memorized how to draw a pelican on a bike and a model that has fundamentally improved its understanding of spatial coordinates and SVG syntax. The former is a parlor trick; the latter is a leap in reasoning. By using a grid of 48 combinations, the researchers were able to prove that the models are not simply recalling a specific image, but are instead struggling or succeeding based on their general architectural capabilities.

For practitioners integrating these models into production pipelines, the takeaway is a warning against the siren song of the viral prompt. A single impressive output is not a signal of intelligence; it is often just a lucky draw from a probability distribution. The only reliable way to verify a model's capability is to adopt the grid testing method used in this study. By testing a core subject against multiple similar subjects and varying the conditions across a matrix, developers can distinguish between general competence and overfitting. If a model excels at drawing a pelican on a bicycle but fails miserably at drawing a heron on a scooter, the performance is a result of memorization, not reasoning.

The industry is moving away from a world where a single benchmark score defines a model's value. The new gold standard is distributional consistency, where the quality of the output remains stable regardless of the specific combination of variables.