Building a developer tool today means designing for more than the developer. Coding agents are increasingly part of the interface to the product, reading documentation, invoking CLIs and APIs, modifying configuration, and deciding what tools and workflows to use to achieve tasks. GitHub’s 2024 developer survey found that 99% of U.S. respondents had tried AI coding tools at work, which increasingly makes agent compatiblity a core requirement of your product.
When we created SKILL.md, AGENTS.md, and llms.txt for Quantiles, we needed to know whether coding agents could actually discover those resources, interpret the instructions correctly, and use them to complete real tasks. We started using Quantiles to evaluate how coding agents interacted with our own documentation. That work made it clear that evaluating a coding agent required an approach that was very different from that of evaluating an LLM.
Inside an Agent Trace
One of the biggest changes we had to make was to capture and evaluate the agent’s execution trace. An agent trace is a structured record of all the things that happened during an agent run. Rather than storing only the task input and final output, a trace captures the sequence of operations that produced the result. Depending on the agent and instrumentation, a trace can include:
- Model requests and responses
- Tool calls, arguments, and results
- Retrievals
- Errors and retries
- Timing and token usage
- Context compactions
- Changes to external state
Spans
A trace is typically organized into spans, where each span represents an operation such as an LLM call, tool invocation, retrieval, or validation step. Spans contain attributes such as timestamps, status, inputs, outputs, and operation-specific metadata, and parent-child relationships preserve how those operations fit into the overall execution.
This figure shows an execution trace for a coding agent running and analyzing a Quantiles evaluation, with each child span representing an operation performed during the workflow:
qt run my-eval11.8 sqt show 1 --json1.7 sCapturing Agent Traces
There are several ways to capture execution traces. Some agent runtimes and frameworks provide built-in tracing or telemetry, while others require additional instrumentation. A common approach is to emit telemetry using OpenTelemetry, often with agent-specific semantic conventions like those defined in OpenInference that add standardized attributes for LLM calls, tools, retrievals, and related operations. Traces can be exported to Quantiles or other observability systems for storage and analysis across runs.
When native tracing is unavailable, an evaluation harness can capture many of the same events by wrapping model clients, tool interfaces, subprocesses, agent hooks, and beyond. The important requirements in all cases are to preserve stable identifiers and timestamps, parent-child relationships between operations, tool inputs and outputs (without leaking secrets and other private data), errors and status codes, and enough metadata to reconstruct the observable execution of each trial.
Evaluate at the Right Level
A trace contains several kinds of evidence, and not every metric needs the entire trace. The goal is to grade agent behavior where it can be observed most directly. Some properties require the complete execution trajectory, while others can be determined from a single tool call, error event, final environment state, or final output.
For example, end-to-end measures can show whether the coding agent successfully completed the requested Quantiles workflow, how long it took, how much it cost, and whether it recovered from failures along the way. At the span level, we can examine whether the agent found and used the appropriate Quantiles resources, such as the SKILL.md, llms.txt, or the appropriate documentation, and whether it executed the correct qt commands with valid arguments. Individual events can capture narrower signals such as a failed command, invalid configuration, permission error, or approval request. Across repeated trials, these measurements can show how reliably and efficiently coding agents can use Quantiles, the nature of failures, and where failures consistently occur.
Validating behavior vs. Prescribing a trajectory
Agents can often complete the same task through several valid sequences of actions so requiring one canonical sequence of tool calls can make an evaluation unnecessarily brittle. Anthropic cautions against enforcing specific execution paths when multiple approaches can be correct. Instead, define the constraints the agent must satisfy regardless of the path it takes. These can include permitted tool use, valid arguments, required approvals, retry limits, prohibited side effects, and the expected final state.
Different agent behaviors require different forms of evaluation. Anthropic recommends combining code-based, model-based, and human graders, using each where its form of evidence and judgment is most appropriate.
| Method | Use when | Examples |
|---|---|---|
| Code-based graders | The criterion can be evaluated deterministically from observable evidence | Schema validity, tool arguments, exit codes, file changes, environment state, required operations, exact outputs |
| Model-based graders | The criterion requires semantic or qualitative judgment that cannot be reliably expressed as deterministic checks at scale | Instruction following, relevance of selected documentation, quality of a response, appropriate handling of ambiguity |
| Human graders | The behavior is nuanced, novel, high-stakes, or requires review or calibration of automated graders | Edge cases, unexpected valid solutions, subjective quality judgments, disagreements about rubric interpretation |
Interpret Traces Carefully
Traces provide a large amount of detailed insight into an agent’s execution, but they have limits. What appears in a trace depends on how the system is instrumented, and recorded activity doesn't always establish the agent’s reasoning or the final state of the environment. Keep these guidelines in mind when using traces for evaluations:
1. Treat traces as observations, not complete records.
A trace can only show what the instrumentation records. Sampling, telemetry limits, unsupported operations, and instrumentation failures can leave gaps in the execution history.
2. Separate execution from reasoning.
A trace can show what the agent did, but since model calls sometimes only show reasoning summaries, a trace can't always tell you why. Base evaluations on behavior you can observe rather than assumptions about the agent’s reasoning.
3. Validate outcomes against the environment.
A trace can show that the agent took the right actions without proving that the task actually succeeded. When possible, verify the final environment state. Doing so might require customized tests, filesystem checks, database queries, or API responses.
4. Allow more than one valid path.
Focus on whether the agent met the task requirements rather than whether it followed one specific path, unless the path itself is part of the success criteria.
5. Keep the measurement system stable.
Keep the evaluation setup consistent and versioned across runs. Changes to the runtime, tools, resources, or instrumentation can change the trace even when the agent’s behavior hasn't changed.
Keep in mind that OpenTelemetry SDKs can limit or sample the data retained for a span. When one of those things happen, traces can be left with gaps, even if they're not readily apparent.
Conclusion
Developer products are increasingly being used by agents, not just people. That means testing them also requires understanding whether agents can navigate their resources, use their tools, recover from problems, and complete intended workflows reliably. Execution traces make those behaviors observable, while validations confirm whether the task actually succeeded. Together, they provide evidence of both the outcome and the path the agent took to reach it.
