Many AI engineers begin their journey into Retrieval-Augmented Generation by jumping straight into the deep end of vector databases and high-dimensional embeddings. The allure of semantic search is strong, but this approach often leads to an immediate collision with reality: unexpected latency, spiraling API costs, and the operational nightmare of re-indexing millions of documents every time a model version updates. The industry is currently seeing a quiet return to fundamentals, where the most resilient pipelines are not those that start with the most complex tech, but those that scale their retrieval strategy in lockstep with actual user failure points.

The Technical Spectrum of Retrieval Costs and Latency

The most streamlined entry point for any RAG system is the BM25 algorithm, the industry standard for full-text search utilized by tools like Elasticsearch and Postgres. This approach operates with a level of efficiency that modern embedding models cannot match, typically recording latency of under 10ms. Because it relies on keyword frequency and document length rather than vector space, it requires no complex chunking strategies, allowing developers to process entire documents as they are. Most importantly, BM25 incurs zero API costs and carries no risk of performance degradation when the underlying LLM is updated.

As the need for better intent recognition grows, the next logical step is the introduction of query rewriting. By utilizing a model like GPT-4o-mini, a system can transform a user's colloquial question into a set of optimized keywords for BM25. This process involves stripping stop words, adding synonyms, and converting domain-specific jargon into terms the search engine can actually find. This layer adds a marginal cost of approximately 0.001 dollars per query but provides a critical advantage: the search strategy can be pivoted instantly by modifying the system prompt, without touching the underlying data index.

When keyword matching fails to capture the nuance of a query, hybrid search becomes necessary. This architecture typically employs a model such as `text-embedding-3-small`, which is priced at 0.02 dollars per 1 million tokens. In a typical reranking pipeline where 50 documents averaging 500 tokens each are embedded in real-time, the cost per query sits at roughly 0.0005 dollars. However, this precision comes with a performance tax, adding 200 to 500ms of latency to the request. This delay is often the primary bottleneck in user experience, forcing a choice between the speed of a keyword search and the intelligence of a semantic one.

For highly complex requests, the architecture evolves into Agentic RAG. This mechanism decomposes a single, multi-intent query into several sub-queries. For instance, a request to read a CSV, clean the data, and plot the results is split into three independent operations executed in parallel. While this increases the query rewriting cost to approximately 0.005 dollars, the resulting precision in retrieval is significantly higher because each sub-query is surgically targeted.

The Friction Between Data Volatility and Infrastructure

The real divergence in RAG strategy appears when deciding how to handle the embedding pipeline. The choice between on-the-fly embedding and full pre-embedding is not a matter of preference, but a calculation of data volatility and query volume. On-the-fly embedding, where documents are vectorized at the moment of the query, is the only viable path for high-volatility environments. If a dataset changes by more than 10 percent daily, such as in news feeds or social media streams, the overhead of constant re-indexing becomes unsustainable.

Conversely, for stable corpora where data changes by less than 5 percent per month and query volume exceeds 10,000 requests per day, full pre-embedding combined with Approximate Nearest Neighbors (ANN) search is mandatory. This setup ensures latency remains under 50ms by shifting the computational burden from the query phase to the ingestion phase. To balance these two extremes, some teams implement a hot/cold tier structure, where frequently accessed documents are pre-embedded for speed, while the long tail of the archive is handled on-the-fly.

This architectural choice also dictates the system's long-term maintenance risk. The transition between embedding models, such as moving from `text-embedding-ada-002` to `text-embedding-3`, reveals a stark contrast in operational fragility. In a full pre-embedding system, a model update requires the entire vector database to be wiped and millions of documents to be re-embedded, a process that can take days and cost thousands of dollars. In an on-the-fly system, the same update is achieved by changing a single line of code specifying the model name.

For teams without deep machine learning expertise, the 80/20 rule applies heavily to RAG. A combination of full-text search and query rewriting often solves 60 percent of all use cases. The temptation to implement hybrid search or ANN from day one often introduces unnecessary complexity that obscures the actual problem: whether the failure is due to the retrieval of the wrong document or the LLM's inability to synthesize the correct one.

Practical implementation should follow a sequential path of optimization. Developers should first establish a baseline with BM25 and collect user feedback for two to four weeks. Only when users report that existing documents are not being found should query rewriting be introduced. Only when the retrieved documents are present but the results are deemed unsatisfactory should the team move toward hybrid search and full pre-embedding.

This disciplined escalation ensures that the infrastructure complexity never exceeds the actual needs of the user base.