Every developer has experienced the moment of disillusionment with AI coding assistants. It usually happens the second the AI moves from a clean, isolated LeetCode-style problem to a sprawling, five-year-old enterprise codebase. In the sandbox, the AI is a virtuoso; in the production environment, it becomes a liability, suggesting changes that break legacy dependencies or hallucinating functions that do not exist. This gap between synthetic proficiency and production utility has remained a blind spot in AI evaluation until now.
The Architecture of Real-World Complexity
Real-SWE emerges as a rigorous response to this discrepancy, shifting the evaluation of AI coding models away from synthetic datasets and toward licensed, private production codebases from actual companies. Unlike previous benchmarks that rely on expert-generated tasks or synthetic scenarios, Real-SWE utilizes verified work assignments that real engineers have already solved. The selection process for these codebases is intentionally stringent, focusing on companies with high traffic, sophisticated engineering teams, and demanding production workloads. This ensures that the AI is not just writing code, but navigating the architectural debt and operational constraints of a living system.
The benchmark measures models across two primary axes: the nature of the coding artifacts and the specificity of the instructions. In a production setting, an agent cannot simply write a function in a vacuum. It must understand the existing architecture, preserve behaviors that users rely on, and operate within strict operational limits. The instructions provided in Real-SWE are designed to mimic real-world tickets; they describe the necessary changes but require the agent to discover the implementation details by exploring the codebase and its surrounding tools. While these prompts are slightly underspecified—similar to the levels found in DeepSWE and Terminal Bench—they provide enough detail to be actionable without handing the model the answer.
The scale of these tasks dwarfs traditional benchmarks. A typical Real-SWE instruction spans 1,742 characters, reflecting the nuance of corporate requirements. More tellingly, the average task involves 11 different files, a significant jump from the 6 files typically seen in FrontierCode or DeepSWE. This increased surface area forces the model to maintain a larger context window and manage complex inter-dependencies. The results are sobering: 71.4% of executions lasting less than 10 minutes failed, while those with longer execution times saw a failure rate of 73.4%. This suggests that simply giving a model more time to think does not solve the fundamental struggle of triaging multiple systems and grasping implicit business logic.
The Enterprise Gap: Context Over Syntax
To understand why these models collapse, one only needs to look at the benchmark's case study on invoice billing and tax settlement systems. In this scenario, the AI is tasked with modifying a billing service to ensure correct tax application while exempting tax-free customers. On the surface, this sounds like a simple conditional logic problem. In reality, it is a nightmare of edge cases. Each company in the platform maintains its own tax rates, uses different destination-based pricing via tax authority providers, or opts out of collection entirely.
The complexity deepens when the agent must interact with external systems. Destination pricing requires managing two separate addresses, priced lines, and product categories, while routing requests to either sandbox or production authorities depending on the account. If an authority rejects an address, the system must report the error without halting the invoice issuance. Furthermore, the AI must ensure that tax rates, totals, and VAT registration numbers for European parties are correctly displayed and that sales are reported to authorities using specific identifiers to ensure ledger reconciliation. All of this requires the agent to correctly utilize environment variables and endpoints such as `TAX_JAR_URL`, `PROD_TAX_JAR_URL`, and `INFLUX_URL`.
This is where the twist occurs: the AI fails not because it cannot write the code, but because it cannot perform the engineering. The evaluation uses a native harness where each agent runs in an isolated sandbox, exposing only the services necessary for the workflow. The scoring is handled by validators inspired by the codebase's own existing test suites. The failure analysis reveals a recurring pattern of cognitive shortcuts. Models frequently build based on guesses about the system rather than verifying the actual state of the workspace. They often miss specific behavioral requirements or, perhaps most dangerously, implement a correct idea but connect it incorrectly to the surrounding systems.
Many models treat the task like a one-off script, placing changes in areas of the code that the running application never actually calls. This reveals a fundamental lack of understanding regarding how software is actually executed in a production environment. The AI is optimizing for the local logic of the function it is writing, while the actual requirement is the global stability of the system. The failure is not one of syntax, but of systemic awareness.
The transition from AI that can code to AI that can engineer requires a shift in how we measure intelligence. Real-SWE proves that the ability to solve a puzzle is not the same as the ability to maintain a product.
