Modern AI development teams are currently trapped in a cycle of framework experimentation. One week, a team might deploy a complex RAG pipeline using LlamaIndex for its superior data connectors. The next, they might migrate to LangGraph to handle sophisticated multi-step orchestration, or adopt the OpenAI Agents SDK to align with corporate standardization. In some cases, they might even mix and match, utilizing Google ADK for multi-agent collaboration or the Claude Agent SDK to leverage native Anthropic capabilities. While this flexibility allows developers to pick the best tool for the job, it creates a hidden and costly tax: the evaluation pipeline.
Traditionally, evaluating an AI agent requires deep integration with the specific SDK being used. Evaluation scripts are often hard-coded to recognize the unique response objects, function call signatures, and tracing patterns of a particular framework. The moment a developer switches from LlamaIndex to LangGraph, the evaluation pipeline breaks. The metrics for accuracy, latency, and goal completion suddenly stop reporting because the system no longer recognizes how the agent is communicating. This creates a perverse incentive where teams stick with inferior frameworks simply because the cost of rewriting their evaluation infrastructure is too high. The industry has reached a point where the diversity of agent frameworks has far outpaced the evolution of the tools used to measure them.
The Architecture of Framework-Agnostic Observability
Amazon Bedrock AgentCore Evaluations addresses this fragmentation by implementing a decoupling strategy that separates the evaluation logic from the underlying framework. Instead of building a unique connector for every SDK, the service relies on a standardized observability layer provided by the Amazon Bedrock AgentCore runtime. This runtime environment handles the critical infrastructure—hosting, scaling, memory management, and observability—allowing the evaluation service to treat the agent as a black box that emits standardized telemetry rather than a specific piece of software written in a specific SDK.
At the heart of this system is OpenTelemetry, a vendor-neutral framework designed to standardize traces, metrics, and logs across distributed systems. By leveraging the AWS Distro for OpenTelemetry (ADOT), AgentCore captures the execution history of an agent as a series of spans. These spans are transmitted via the OpenTelemetry Protocol (OTLP) and routed directly into Amazon CloudWatch. This architecture ensures that as long as the agent framework supports OpenTelemetry—which most major libraries now do either natively or through community plugins—the evaluation service can ingest the data regardless of whether the agent was built with Strands Agents, LangGraph, OpenAI Agents SDK, LlamaIndex, Google ADK, or the Claude Agent SDK.
By shifting the focus from SDK-specific hooks to infrastructure-level telemetry, Amazon Bedrock AgentCore removes the need for developers to manually write data collection code every time they update their tech stack. The runtime automatically injects the necessary tracing context, ensuring that the evaluation pipeline remains intact even if the entire agent logic is swapped from one framework to another. This transforms evaluation from a fragile, manual process into a persistent utility that exists independently of the agent's implementation details.
The Semantic Bridge and the Three-Span Logic
The real technical challenge in framework-agnostic evaluation is not just collecting data, but making sense of it. Different frameworks describe their actions differently. Some might call a tool execution a function call, while others call it a tool invocation. To solve this, AgentCore Evaluations implements a semantic bridge that integrates two distinct schemas: the OpenTelemetry GenAI semantic conventions and the OpenInference specifications. OpenTelemetry GenAI defines operational units like chat, embeddings, and execute_tool, while OpenInference categorizes spans into types such as LLM, TOOL, RETRIEVER, and GUARDRAIL. AgentCore bridges these two standards, converting disparate naming conventions into a single, unified data structure that the evaluator can process.
To maintain this consistency without becoming overwhelmed by the noise of complex agent traces, the service employs a strict three-span role identification mechanism. Out of the hundreds of spans generated during a complex agent session, the evaluation service identifies and extracts only three critical roles: the Input span, which contains the user's original request; the Tool Call span, which records the agent's interaction with external functions; and the Output span, which captures the final response delivered to the user. All other telemetry—such as retrieval steps, reranking processes, guardrail checks, and memory retrievals—is treated as supplementary context. This design ensures forward compatibility; if a new framework introduces a new type of span for a novel AI capability, the evaluation service will simply treat it as context rather than crashing due to an unrecognized data format.
This recognition process is governed by a strict `scope.name` validation mechanism. The system automatically identifies instrumentation libraries that begin with the prefixes `opentelemetry.instrumentation.*` or `openinference.instrumentation.*`. If a developer uses a custom tracing library with a non-standard prefix, such as `mycompany.agent.tracing`, the system intentionally ignores those spans even if the internal structure follows the standard. This constraint is a deliberate guardrail against data pollution, ensuring that only libraries that explicitly declare compliance with global semantic standards are used to calculate performance scores.
For a successful evaluation run, two operational prerequisites must be met. First, the agent's spans must possess a `session.id` attribute that matches the `runtimeSessionId` used during the call. While the AgentCore runtime and ADOT handle this injection automatically, developers using custom environments must ensure this alignment to allow the service to group spans into a coherent session. Second, the data source must contain the actual message content, not just the metadata of the span. This requires that the agent's log groups are correctly configured alongside the `aws/spans` log group in CloudWatch.
Once these conditions are met, the system applies a standardized set of metrics across all frameworks. These include the GoalSuccessRate to measure task completion, Correctness to verify factual accuracy, and Helpfulness to assess the quality of the interaction. Additionally, the system supports custom LLM-as-a-judge metrics, allowing teams to define their own success criteria. Detailed information on the compatible ecosystem can be found in the Supported agent frameworks documentation.
By decoupling the measurement of success from the tools used for construction, Amazon Bedrock AgentCore effectively lowers the cost of innovation. Developers are no longer penalized for experimenting with new frameworks, as the objective truth of their agent's performance is now anchored in a standardized telemetry layer rather than a fragile set of SDK-specific scripts.
