The current era of LLM orchestration is defined by a relentless pursuit of the optimal cost-to-performance ratio. Engineering teams are increasingly deploying routing layers to intercept prompts and steer them toward the cheapest possible model capable of handling the task. The logic is seductive: send a simple greeting to a lightweight model and reserve the frontier models for complex reasoning. This strategy promises a drastic reduction in monthly API bills without sacrificing quality, turning the LLM gateway into a sophisticated traffic controller.
The Mechanics of the Routing Failure
Manifest, an LLM gateway service, leaned heavily into this philosophy when it launched its core LLM router feature in March. The system was built on a four-tier complexity classification framework: simple, standard, complex, and reasoning. By categorizing every incoming request into one of these buckets, the router could dynamically assign the most cost-effective model for the job, theoretically eliminating the waste of using a high-cost model for a trivial task.
However, the operational reality diverged sharply from the theoretical model. By June, the team reached a critical decision to discontinue the feature, leading to its complete removal from the platform on September 1. The primary driver for this reversal was the overwhelming impact of cache efficiency. In modern LLM architectures, cache read costs are significantly lower than non-cached input costs, typically ranging from 75% to 90% cheaper.
Because the Manifest router was designed to optimize for cost, it naturally factored these cache savings into its decision-making process. This created a feedback loop known as model stickiness. Once the router selected a model for a specific context, the massive cost advantage of the existing cache made that same model the most economical choice for all subsequent related queries. The router stopped routing. It effectively became a static assignment tool that clung to the first model it picked, rendering the complex routing logic and the four-tier classification system entirely redundant.
The Hidden Cost of Dynamic Routing
Beyond the technical failure of cache-aware routing, a deeper friction emerged between automated orchestration and the actual nature of AI development. The fundamental flaw in prompt-based routing is the assumption that a prompt contains all the information necessary to determine a task's complexity. In practice, a prompt is merely a trigger. The true complexity of a task often remains hidden until the execution phase, specifically during tool calls or web search operations.
Consider a request to improve a software repository. The initial prompt might look simple, but the actual difficulty depends entirely on the repository's architecture, the size of the codebase, and the intricacy of the dependencies. A router analyzing only the prompt cannot predict these variables. Consequently, the system frequently assigned an underpowered model to a task that only revealed its true complexity halfway through execution, leading to failure or degraded output.
This instability introduced a new layer of operational overhead that eclipsed the token savings. To make a router reliable, engineers must invest heavily in system prompt tuning, rigorous evaluation frameworks, and advanced observability to track why a specific path was chosen. In autonomous agent workflows, a single routing error can trigger a cascade of failures across the entire pipeline. The human cost of managing this uncertainty—the hours spent debugging routing logic and refining evals—proved to be far more expensive than the inference costs the router was intended to save.
Furthermore, the constant switching of models during a session disrupted the user experience. Professional engineers do not just use models; they master them. Every model has specific nuances, strengths, and failure modes. When a gateway silently swaps models mid-stream, it strips the developer of the ability to build a mental model of the tool's behavior. The loss of predictability and control became a liability that outweighed the marginal financial gain of using a smaller model for a few prompts.
The industry is discovering that for most production use cases, the most efficient path is not a complex web of routed models, but a single, proven model that provides consistent and predictable results.
