Enterprise AI developers are currently trapped in a high-stakes balancing act between recall and budget. To ensure a Retrieval-Augmented Generation (RAG) system doesn't miss the critical piece of information needed for an accurate answer, the standard industry practice is to cast a wide net. This usually means retrieving between 5 and 20 text chunks per query. While this high-recall strategy provides a safety net against information loss, it imposes a heavy token tax. Every irrelevant sentence retrieved is a token paid for, and as workloads scale to millions of queries, these redundant tokens transform from a minor overhead into a massive operational burden. The tension is clear: developers want the reliability of a wide search but cannot afford the cost of processing the resulting noise.
The Architecture of Query-Aware Compression
Query-aware compression introduces a strategic filtering layer between the retrieval engine and the final generation model. Instead of piping every retrieved chunk directly into the primary LLM, this pattern uses a smaller, more efficient model to act as a precision filter. This filter examines each chunk against the user's specific query and extracts only the verbatim spans of text that contribute to the answer, discarding the surrounding noise. By refining the context before it reaches the expensive primary model, the system drastically reduces the absolute volume of input tokens.
This orchestration is built on Amazon Bedrock, utilizing a serverless pipeline centered around AWS Lambda. The process begins with Amazon Bedrock Knowledge Bases, which leverages Amazon OpenSearch Serverless to convert user queries into vectors and retrieve the top-k most similar text chunks. These chunks are then passed to a single AWS Lambda function that manages the two-stage model call. To ensure consistency and prevent the filter model from inventing information, Claude Haiku is employed as the compression model with the `temperature` parameter strictly set to 0.0. This deterministic setting ensures that the model does not paraphrase or summarize, but instead performs exact extraction of the original text.
To maintain a clean interface, the system utilizes the Amazon Bedrock Converse API. This API provides a unified way to interact with different models, allowing developers to switch between model versions or families without rewriting the core integration logic. The Lambda function first calls Claude Haiku to strip away irrelevant content and then immediately passes the refined, high-density context to the primary model for the final response. By pairing a small model and a large model from the same family, the system minimizes discrepancies in linguistic understanding and maximizes compatibility.
The Trade-off Between Latency and Truthfulness
The impact of this compression layer is most evident when comparing the baseline RAG performance against the optimized pipeline. Simple query-aware compression reduces operational costs by 33% and slashes the number of input tokens by 8.6 times. When this is combined with a Rerank step—which re-evaluates the relevance of retrieved documents before they even reach the compression model—the efficiency gains climb further. The combined Rerank and compression approach reduces costs by 36% and decreases input tokens by a staggering 10.1 times.
Beyond the financial metrics, the most significant shift occurs in the reliability of the output. In the baseline configuration, the hallucination rate—the frequency with which the model generates claims not supported by the source documents—stands at 51%. Implementing simple compression drops this rate to 44%, and adding Rerank further suppresses hallucinations to 38%. This suggests that providing a model with too much irrelevant information actually increases the likelihood of error. When the primary model is forced to sift through a mountain of noise to find a needle of truth, it is more prone to interference and confusion. By increasing the information density of the prompt, the compression layer effectively narrows the window for potential hallucinations.
However, this efficiency comes with a specific set of trade-offs. Adding a secondary model call inevitably increases the initial latency of the request. Yet, this is partially offset by the fact that the primary model processes significantly fewer tokens, which speeds up the actual generation phase. Analysis using an LLM-judge to score responses on a scale of 1 to 5 reveals that correctness remains stable, with a variance of less than 0.07 compared to the baseline. While the accuracy holds, there is a slight decline in completeness and citation accuracy, as some peripheral context is lost during compression. Conversely, conciseness improves, as the model no longer spends tokens on redundant phrasing or filler content.
For developers implementing this in production, the deterministic nature of the filter is non-negotiable. The prompt must explicitly forbid paraphrasing to ensure that the primary model receives original source text, which is critical for maintaining the integrity of citations. While the cost savings are immense for large-scale corpora, the decision to implement this layer must be weighed against the specific needs of the user experience. In real-time chat applications where every millisecond counts, the added latency of the Claude Haiku call must be measured against the token savings to ensure the trade-off remains positive.
This shift toward query-aware compression signals a broader evolution in RAG design, moving away from the brute-force approach of maximizing context toward a philosophy of maximizing signal-to-noise ratios.
