Imagine a high-stakes automotive marketplace during peak hours, where 1,500 concurrent users are hunting for specific vehicle inventory. In this environment, a dealer isn't just chatting with a bot; they are attempting to execute a business transaction. When an AI agent misinterprets a search query or selects the wrong tool to filter a database, the result isn't just a slightly awkward conversation—it is immediate user churn. For Motorway, the gap between a functional agent and a reliable one was a glaring error rate of one in eight requests. In a professional B2B context, a 12.5 percent failure rate is an existential risk to user trust.

The Architecture of a High-Precision Search Agent

To bridge this reliability gap, Motorway engineered a sophisticated search agent by integrating the Strands Agents SDK with Amazon Bedrock AgentCore. This setup leverages Amazon Bedrock AgentCore as a fully managed service to deploy and operate agents at scale. The agent is tasked with navigating a complex dataset of vehicle attributes, utilizing eight distinct tools to filter over 89 different vehicle properties and perform vector similarity searches. The technical backbone relies on LanceDB for vector storage and Amazon Titan Text Embeddings V2 to convert natural language into precise numerical representations. At the center of the reasoning process is Claude, which analyzes user intent and orchestrates the selection of the appropriate tools.

The operational flow begins when a dealer submits a natural language query through a web interface. This request hits the Amazon Bedrock AgentCore Runtime, which acts as the conductor for the eight available tools. The runtime uses Claude for high-level reasoning and Amazon Titan for embedding processing to determine the exact sequence of tool executions. Once the tools return the necessary data, the runtime synthesizes the final result for the dealer. This pipeline is designed to transform ambiguous human requests into rigid data queries, minimizing the risk of missing relevant vehicles in the inventory.

However, Motorway recognized that a single successful response is a vanity metric. In customer-facing agents, the critical metric is the probability of consecutive success, denoted as pass^k. If an agent has a 75 percent success rate for a single turn, the probability of it succeeding across three consecutive interactions drops precipitously to approximately 42 percent. Because users expect a consistent baseline of quality across an entire session, Motorway implemented a three-layer evaluation pipeline to enforce a strict error rate of one in fifty.

Moving Beyond Textual Benchmarks to Functional Trajectories

Most LLM evaluations treat the model like an engine on a test bench, measuring consistency, factual accuracy, and relevance of the final text. While this works for chatbots, it fails for agents. A text-based metric can tell you if an agent's response sounds polite and professional, but it cannot tell you if the agent actually called the correct search tool when a user asked for a Grade 1 Suzuki. If the agent produces a natural-sounding answer but used the wrong filter parameters in LanceDB, it is a functional failure, regardless of how fluent the prose is.

Motorway shifted the focus from output quality to trajectory validation. This is the difference between measuring an engine's horsepower and conducting a road test. The system must verify that the agent maintained context across multi-turn conversations and passed the exact required parameters to the database. For instance, a structured query like Volkswagen Golf 7-12 years old is easy for most systems. But a colloquial request such as I’m looking for an older VW tests the semantic search layer. If the agent fails to translate older VW into a specific year-range filter, the search fails. Textual evaluation would only see a polite response; trajectory evaluation sees a broken tool call.

To solve this, Motorway introduced the pass^k metric to combat the inherent volatility of LLMs. Since a model might succeed once by chance, the system now requires success across multiple trials before a version is cleared for deployment. This is implemented via the `run_all_layers` function, which automates multi-trial testing:

run_all_layers(task_fn, registry, num_trials=5)

By repeating the same task five times, the pipeline ensures that performance is a result of robust reasoning rather than a lucky seed. To further harden the system, Motorway expanded its test suite from 50 to 150 cases over three months, incorporating real-world interaction data. They specifically added negative cases—scenarios where the agent must explicitly avoid calling certain tools. For example, if a user is simply asking for information, the agent must not trigger a payment or modification tool. To simulate complex human behavior, they integrated ActorSimu, which acts as a virtual user to test multi-turn dialogue consistency and prevent context collapse.

The Three-Layer Deployment Gate

To ensure no regression reaches production, Motorway utilizes a three-layer evaluation framework that acts as a deployment gate during the GenAIOps build phase. Each layer has a specific threshold that must be met for the agent to pass.

The first layer is Tool Usage, which requires a pass rate of 95 percent or higher. This layer uses deterministic, code-based graders—specifically the `ToolSelectionGrader` and `TrajectoryOrderGrader`—to ensure that the same input always results in the same tool selection and calling sequence. This removes the randomness from the most critical part of the functional chain.

The second layer is Reasoning, with a threshold of 85 percent. Here, the system employs an LLM-as-judge approach using the `HelpfulnessEvaluator` and `TrajectoryEvaluator` from the strands-agents-evals framework. Instead of just checking the final answer, the judge analyzes the agent's internal thought process. This prevents the agent from being rewarded for arriving at the correct answer through flawed logic.

The final layer is Output Quality, requiring a 90 percent pass rate. Using the `OutputEvaluator` and `GoalSuccessRateEvaluator`, the system determines if the response is actually useful, factually accurate, and actionable for the user. This layer ensures the final delivery matches the user's requirements in a polished format.

For B2B implementations, the cost of this rigor is minimal compared to the risk. Running a sample evaluation set on Amazon Bedrock costs between 5 and 10 dollars. In a high-ticket industry like automotive sales, the cost of a few inference calls is negligible compared to the potential loss of a high-value transaction caused by an AI hallucination. Security is handled via least-privilege IAM roles and the AWS Systems Manager Parameter Store for API key management, while strictly typed parameters prevent prompt injection attacks.

This rigorous cycle of using production data to create new test cases ensures that the agent evolves alongside the users. By prioritizing tool call precision over linguistic flair, Motorway has turned a volatile AI experiment into a reliable business tool.