The modern AI engineer spends a disproportionate amount of time acting as a digital detective. When an agent's success rate unexpectedly dips from 85% to 70%, the remedy is rarely a simple prompt tweak. Instead, it involves a grueling descent into the logs, manually scanning through hundreds of execution spans to pinpoint exactly where the logic diverged. This manual audit is the primary bottleneck in the agentic development lifecycle, turning what should be a rapid iteration cycle into a slow process of trial and error.
The Architecture of Automated Failure Classification
Strands Evals addresses this inefficiency through its Detector, a specialized diagnostic engine designed to replace manual log analysis with structured automation. Rather than providing a binary pass-fail grade, the Detector employs a comprehensive taxonomy of nine parent categories to identify the nature of a failure. These categories include hallucination, incorrect actions, orchestration errors, task instruction non-compliance, execution errors, context handling errors, repetitive behavior, LLM output issues, and configuration mismatch. By categorizing errors this way, the system can distinguish between an agent that fabricated information and one that simply failed to follow a specific tool parameter.
The output of this process is not a vague summary but a set of actionable data points. For every identified error, the Detector provides the precise span location where the failure occurred, the matched category, a confidence score for the diagnosis, and the actual evidence sentence extracted from the trace. This allows developers to bypass the entire log history and jump directly to the point of failure. When a system shows a high volume of execution errors, the developer knows to refine the tool definitions; when task instruction non-compliance dominates, the focus shifts to sharpening the system prompt. This precision transforms debugging from a guessing game into a targeted engineering task.
From Symptom Detection to Causal Intelligence
While many evaluation tools can tell a developer that a system failed, the critical gap remains the distinction between a symptom and a cause. Strands Evals bridges this gap using a two-stage analysis pipeline. The first stage, failure detection, scans every span within a session against the nine-category framework to flag anomalies. However, the real insight happens in the second stage: root cause analysis. In complex agentic workflows, a single early mistake often triggers a cascade of subsequent failures. To solve this, the Detector tracks the causal chain and assigns a hierarchy to each failure, labeling them as PRIMARY, SECONDARY, or TERTIARY.
This hierarchical approach prevents developers from wasting time fixing secondary symptoms. If an agent hallucinates a value because a previous tool call returned a malformed response, the hallucination is a secondary symptom, while the tool failure is the primary cause. The system then maps these root causes to specific remediation strategies. A failure in parameter validation due to a schema error is flagged as a `TOOL_DESCRIPTION_FIX`, whereas a hallucination occurring after a tool failure is flagged as a `SYSTEM_PROMPT_FIX`. This direct mapping from error to solution drastically reduces the risk of introducing new bugs while trying to fix old ones.
To handle varying session scales, the Detector employs three distinct processing strategies to balance precision and cost. Small sessions are analyzed directly. Medium-sized sessions undergo path pruning, where only ancestor and descendant spans are retained to maintain context. For massive sessions that would exceed LLM context windows, the system uses chunked analysis, dividing the trace into overlapping windows and merging the results. This ensures that the diagnostic quality remains consistent regardless of the complexity of the agent's trajectory.
This creates a fundamental separation between the Evaluator and the Detector. The Evaluator focuses on the how well—calculating goal achievement rates, tool selection accuracy, and helpfulness scores. It acts as a signal light for deployment. The Detector focuses on the why—analyzing individual spans and causal chains to provide a blueprint for improvement. By decoupling performance measurement from root cause diagnosis, Strands Evals eliminates the debugging bottleneck that typically follows a failed test run.
Integrating this diagnostic power into a production pipeline is handled through a streamlined API. Developers can pass a session object to the `detect_failures` function to immediately retrieve failure locations and confidence scores:
detect_failures(session)To move deeper into the causal chain, the `analyze_root_cause` function separates the symptoms from the source and suggests whether the fix belongs in the system prompt or the tool description:
analyze_root_cause(failures)For a comprehensive end-to-end workflow, the `diagnose_session` function wraps both detection and root cause analysis into a single pipeline, returning a consolidated `DiagnosisResult`:
diagnose_session(session)For those integrating these tools into CI/CD pipelines, the `DiagnosisConfig` allows for strategic cost management. The `ON_FAILURE` mode ensures that the Detector only runs when the Evaluator returns `test_pass=False`, optimizing LLM token spend. Conversely, the `ALWAYS` mode analyzes every case, which is invaluable for identifying inefficient paths that technically passed the test but are suboptimal for production.
For practitioners operating within the Amazon Bedrock ecosystem, the Detector is specifically optimized for Bedrock and Strands Agents. Implementation requires OpenTelemetry tracing to be enabled, ensuring that agent sessions can be exported in JSON format. For existing agents, the `CloudWatchProvider` can be used to fetch stored trace data, a process detailed in the Strands Agents SDK User Simulation guides. Because the Detector utilizes LLMs for its analysis, the `ON_FAILURE` configuration is recommended to manage Amazon Bedrock costs effectively. Furthermore, infrastructure design should account for the agent's average trace length to ensure that the pruning and chunking strategies align with the chosen model's context window.
The transition from intuition-based debugging to an automated diagnostic pipeline marks a shift in how AI agents are matured. By replacing the fatigue of manual log diving with a structured, causal analysis, the distance between detecting a performance drop and deploying a fix is reduced from hours to minutes. The goal is no longer just to know that the agent failed, but to have the exact coordinates for the repair.
