The AI industry is currently locked in a war of attrition against the laws of physics. Every new model release promises a larger context window, the ability to ingest entire libraries, or the capacity to reason through complex codebases in a single pass. But behind the curtain of these capabilities lies a brutal mathematical reality: the more we ask these models to remember, the more the power grid screams. The pursuit of artificial general intelligence has evolved into a struggle to manage the sheer heat and electricity required to keep the current architectural standard alive.

The Mathematical Tax of Dense Attention

The engine driving this entire era is the Transformer. Justin Dangel, the CEO and co-founder of Subquadratic, notes that the modern AI industry is essentially a monument to this single architecture. Since its inception nearly a decade ago, the Transformer has become the standard blueprint for every commercial AI service in existence. However, the cost of this dominance is staggering. When a Transformer processes a document of 10,000 words, it does not simply read the text; it performs approximately 50 million multiplication operations.

This computational explosion is the direct result of the dense attention mechanism. In this structure, the model encodes the meaning of a text block by comparing every single token in a sequence against every other token through multiplication. Because every word must attend to every other word, the computational load does not grow linearly as the text gets longer—it grows quadratically. This inefficiency is why OpenAI President Greg Brockman recently indicated that the company expects to spend 50 billion dollars on computing this year. The financial expenditure is a proxy for a physical crisis. The International Energy Agency has already predicted that total power consumption for data centers will double by 2030. As parameter counts climb, the cost of cooling and electricity ceases to be a line item on a balance sheet and becomes a hard physical limit on how much a model can actually scale.

The Illusion of Progress in LLMs+

For several years, the prevailing belief in Silicon Valley was that scaling laws would solve these inefficiencies. The assumption was that more data and more GPUs would eventually brute-force the Transformer's limitations into insignificance. But the industry has reached a point of diminishing returns where the architecture itself is the bottleneck. This has led to the emergence of what is being called LLMs+. These are not new architectures that replace the Transformer, but rather extended models that use sophisticated workarounds to patch its fundamental flaws.

When we see recent breakthroughs in reasoning capabilities or the ability to handle massive input volumes, we are often seeing the results of these patches rather than a fundamental evolution of the engine. LLMs+ attempt to simulate a larger context window or better memory management, but they do so by layering fixes over the original dense attention structure. This creates a critical tension in the development of autonomous agents. For an agent to be truly effective, it must often take the output of one LLM and use it as the input for another, creating a chain of information. When the underlying architecture struggles to maintain a stable context window due to computational costs, the agent hits a cognitive ceiling. It cannot effectively audit a massive software repository or manage a complex multi-step workflow because the cost of maintaining that information exceeds the available hardware resources.

We are currently in a phase where the industry is adding turbochargers to an engine that is fundamentally overheating. The transition from standard LLMs to LLMs+ is an admission that the Transformer's approach to attention is unsustainable for the next leap in intelligence. The focus has shifted from simply making models larger to finding ways to bypass the quadratic cost of the attention mechanism without breaking the model's ability to understand nuance.

The industry is now waiting for the moment when a truly sub-quadratic alternative makes the Transformer's 50 million multiplications look like an ancient inefficiency.