The AI community has long struggled with a recurring paradox: a model that demonstrates brilliance in a chat interface often collapses when placed inside a rigid benchmark harness. Developers frequently witness a frontier model fail a complex reasoning task, only to realize later that the failure was not a lack of intelligence, but a failure of the plumbing. This week, the gap between theoretical capability and measured performance was laid bare through the testing of GPT-5.6 Sol on the ARC-AGI-3 benchmark, revealing that the difference between a mediocre score and a breakthrough often lies in a few lines of API configuration.

The Performance Gap in ARC-AGI-3

ARC-AGI-3 serves as a brutal litmus test for fluid intelligence, requiring AI agents to deduce the underlying logic of 2D puzzle games without any explicit instructions. Unlike standard benchmarks that rely on pattern matching from training data, ARC-AGI-3 measures the ability to learn and adapt in real-time. For context, the average human tester records a Relative Human Action Efficiency (RHAE) score of 48%. Most frontier models struggle to even clear the first level of the six-stage puzzle sequence, often getting trapped in repetitive loops or failing to grasp the basic spatial rules of the environment.

When GPT-5.6 Sol was first deployed using the official, general-purpose harness, it recorded a score of 13.3%. While this was higher than many of its peers, it remained far below human parity and suggested a ceiling in the model's reasoning capabilities. However, the narrative shifted entirely when engineers applied a specific set of API optimizations. By modifying the execution environment rather than the model's weights, the score jumped to 38.3%. This nearly threefold increase proves that the model was capable of solving all six levels of the benchmark, but was being throttled by the way it interacted with the task.

The Architecture of Retained Reasoning and Compaction

The dramatic performance leap was driven by two critical adjustments within the Responses API: the implementation of retained reasoning and the shift to context compaction. To understand why these matter, one must look at how the official harness typically handles state. In a standard setup, the harness deletes the model's private reasoning messages after every action. This forces the model to re-interpret the entire game state from scratch every single turn. While the model can see its previous moves and a brief memory log, it loses the specific logical chain—the a-ha moment—that led to those moves. It is effectively a genius with short-term memory loss, unable to carry a strategic insight from turn five to turn six.

By passing the `previous response ID` through the Responses API, engineers enabled Retained Reasoning. This setting preserves the private reasoning logs as part of the conversation history. Instead of starting from zero, GPT-5.6 Sol can now reference its own internal monologue from previous turns. This eliminates the need for redundant thinking time and allows the model to maintain a consistent long-term strategy. The model no longer just sees that it moved a pixel to the left; it remembers why it decided that moving the pixel to the left was the key to unlocking the puzzle's logic.

Parallel to this, the team addressed the constraints of the 175,000 token context window. The default method for managing overflow is rolling truncation, which simply deletes the oldest messages once the limit is reached. In a complex puzzle, the oldest messages often contain the most critical observations—the initial discovery of the game's rules. When rolling truncation kicks in, the model forgets the fundamental premise of the puzzle and begins exploring paths it has already proven to be failures. Furthermore, operating at the absolute edge of a full context window often introduces subtle computational overhead that degrades reasoning precision.

To solve this, the engineers activated the Compaction setting. Instead of deleting old data, the API now summarizes core context and preserves it in a condensed format. This shift reduced output token usage by 6x compared to the previous method. Because action grid data in ARC-AGI-3 is processed at a 1:1 tokenization ratio, this efficiency is vital. Compaction allows the model to maintain a high-density map of its past attempts and failures without clogging the context window, ensuring that the logic derived in the first ten minutes of a session is still available an hour later.

General Harnesses vs. Optimized Execution Bundles

This discrepancy highlights a fundamental philosophical divide in AI evaluation. The official ARC-AGI-3 harness is designed for universality; it strips away specialized features to ensure a level playing field where model flaws are exposed. This is valuable for academic comparison but irrelevant for production. Commercial developers, conversely, treat the model and its harness as a single execution bundle. The goal is not to see how the model performs in a vacuum, but how it performs when the interface is tuned to its specific quirks.

GPT-5.6 Sol has already demonstrated this synergy in other complex environments. It has successfully navigated Pokémon FireRed using a vision-specific harness and tackled Slay the Spire via a Codex computer-use harness. Perhaps most impressively, it solved random daily challenges in Slay the Spire 2—a game released after the model's knowledge cutoff—proving that an optimized interface can compensate for a lack of pre-existing knowledge by maximizing the model's adaptive reasoning.

For AI engineers, the lesson is clear: a benchmark score is not a measurement of raw intelligence, but a measurement of the combination of model weights, API settings, and harness design. If a model is underperforming, the bottleneck is often the lack of reasoning persistence or inefficient context management. To truly validate a model's reasoning ceiling, engineers should move away from general harnesses and instead implement a system that explicitly passes the `previous response ID` and utilizes summary-based compaction.

Those looking to stress-test their own implementations can utilize the public puzzle sets available at arcprize.org/tasks. By comparing results between a standard rolling-truncation setup and a compaction-enabled setup, the impact of the execution environment becomes immediately apparent. The path to AGI-level reasoning may not just be about building larger models, but about building the sophisticated memory architectures that allow those models to actually use the intelligence they possess.