Many things can go wrong when agents use real products. Commands fail, the right reference is hard to find, dependencies are missing, and the environment doesn't behave as expected. We can't eliminate every failure. What we need to understand is whether agents can recover from them.
To measure an agent's recovery means to determine what happens between a failure and the its return to productive work. These measurements tell us whether agents can resolve problems and continue toward the intended outcome, while execution traces show what that recovery required. Together, these signals help us understand how reliably agents navigate problems and where the product can make that path easier.
Defining agent recovery
Agent recovery happens when an agent resolves or works around a problem that was blocking progress and can continue the task. Along the way, it may read documentation, change a configuration, retry an operation, or take another valid approach. These actions are part of the recovery process, but it is their ability to restore progress that ultimately matters.
We can capture this as a failure episode, which includes the initial failure and the related attempts to resolve it. The episode ends when there's evidence that the agent has restored progress, either by successfully retrying the original operation or by reaching the same intended outcome another way. As Anthropic notes in its guidance on agent evaluation, there may be more than one valid way to complete a task. For recovery, this gives agents flexibility in how they solve the problem while giving the evaluator a clear outcome to verify.
The figure below follows an example recovery episode from initial failure through investigation, correction, and verification.
- 1. Failure detected
Run command→ Required configuration missing→ Failure episode begins - 2. Investigate
Read documentation✓→ Tool call succeeds→ Failure remains unresolved - 3. Retry
Run command✕→ Same configuration error→ Same failure episode - 4. Correct
Update configuration✓→ Required field added→ Resolution candidate - 5. Verify
Run command✓→ Original operation succeeds→ Verified recovery
Measuring verified recovery
Before measuring recovery, we first need to know that a failure actually happened. That's not always obvious from a trace. Some failed operations are expected, while others are handled without interrupting the larger workflow. Looking at each attempt in context during AI agent evaluation helps identify which failures require recovery and how the agent responds to move forward.
The same principle applies to determining whether the agent recovered. Recovery is easier to measure when there is a verifiable outcome. If a test failed, we can check whether it now passes. If an artifact was missing, we can verify that it now exists and is valid. If state needed to change, we can verify the resulting state. This gives us concrete evidence that the agent moved past the original problem and continued making progress.
Calculating recovery rate
Once recovery is verified, calculating the rate is relatively simple.
An eligible failure is one the agent has the resources and opportunity to recover from within the evaluation, given its available tools, permissions, execution budget, and access to human assistance.
Using the same eligibility criteria makes recovery rates easier to compare across agents and trials. Eligible failures remain in the measurement even when the agent does not resolve them before the run ends. This gives us a more complete picture of how consistently the agent can recover with the resources available to it.
Recovery effort
Once we know an agent can recover, the next question is what it needs to do so. Recovery effort metrics capture the additional tool calls, retries, tokens, and time required to move past a failure. These measurements align with OpenAI’s guidance on agent evaluations, which recommends tracking attempts, resource use, and execution time alongside task outcomes. Tracking recovery effort can show where documentation, interfaces, and tooling already support agents well and where they can make successful use of the product even easier.
Measure effort across all eligible failure episodes, not only successful recoveries. Unresolved episodes may consume substantial time, tokens, and tool calls before the run ends.
Multidimensional evaluation
Recovery is best understood as a set of related signals. How often problems arise, whether agents resolve them, what resolution requires, and whether the task ultimately succeeds each reveal something different. Stanford’s HELM research provides a broader precedent for this multidimensional approach to evaluation, where the value comes from seeing the tradeoffs between measurements.
Consider this hypothetical comparison of two versions of the same developer tool (Product A vs Product B) and how a coding agent recovers when something goes wrong:
| Measurement | Product A | Product B |
|---|---|---|
| Summary | Lower recovery rate, but fewer failures, less recovery effort, and more successfully completed tasks. | Higher recovery rate, but more frequent failures, more tool calls to recover, and fewer successfully completed tasks. |
| Tasks | 100 | 100 |
| Eligible failure episodes | 10 | 40 |
| Verified recovered failure episodes | 7 | 32 |
| Verified recovery rate | 70% | 80% |
| Median tool calls to recovery | 2 | 6 |
| Verified successful tasks | 95 | 90 |
Problems are inevitable in real agent workflows, and some will lead to task failure. When recovery is measurable, it becomes another source of information about how agents experience a product, what they understand, where they struggle, and what helps them move forward. That gives teams something more useful than a recovery rate alone. It shows where improvements can help agents recover more efficiently, complete more tasks, and rely less on human intervention.
