An agent needs reliable tools, access to relevant data, and enough context to understand a task and complete it. When we evaluate an agent, those tools, data, and context - along with permissions and the system’s starting state - make up the evaluation environment. The tricky part is making that environment reflect the conditions the agent would encounter in the real world while keeping the test controlled and repeatable. Choices about what to include, simplify, or leave out can shape both the agent’s performance and what the results tell us.
What's an agent evaluation environment?
An evaluation environment is the set of tools, applications, data, documentation, permissions, and initial conditions an agent works with when completing a task. For a coding agent, this might include a repository, a terminal, installed dependencies, a CLI, and product documentation. Less visible details matter too, including the working directory, existing configuration, cached results, and files created by previous commands.
These details shape what your agent evaluation can measure. For example, here's a common agent evaluation task we often test at Quantiles:
Install the Quantiles skill and run an evaluation using the attached dataset of five tasks. Analyze the results and summarize the findings, including any errors encountered.
For this task, the agent needs access to:
- The dataset of five tasks
- A terminal
- Relevant documentation
- Any credentials required to run the evaluation
- A clean starting state for each task
We made the Quantiles skill available to install without telling the agent where to find it, so we could test whether it could discover the skill independently.
What does a realistic evaluation environment need?
A realistic evaluation environment doesn’t need to reproduce an entire production system. The key is knowing what you can simplify without making the task easier or changing what success means. Details that affect the agent’s decisions, the actions it can take, or the consequences of those actions need to remain intact. The following requirements help you preserve those details while keeping results verifiable and the evaluation repeatable.
Use representative tasks and data
Tasks drawn from everyday workflows help you understand how an agent would perform in the real world. The data should include the details and variations needed to test whether the agent makes the correct decision in each case. And sometimes using synthetic or sanitized records can give that realism without using sensitive production data.
Here’s a useful starting mix for tasks:
- Straightforward requests
- Cases that require interpreting a rule
- Situations where the agent needs more information before proceeding
- Support questions
- Onboarding experiences
- Workflows your team already tests
An expected outcome makes it easier to verify whether the agent completed the task correctly. For more exploratory evaluations at Quantiles, we define broader success criteria and leave room for different solutions. That lets us see how Codex, Claude, and other agents approach the same problem.
Give tools realistic behavior
Give the agent tools that let it work through the task as it would in practice. Those tools should validate inputs, return useful responses, and provide clear error messages when a request fails. You can also introduce failures deliberately to see whether the agent can recover. Record which failures you introduced so you can distinguish the conditions you’re testing from unexpected problems in the test setup.
Here's a tool failure we often see at Quantiles, so we've had to add instructions in the skill to make sure agents know how to recover:
qt resume <run_id> --jsonKeep the evaluation configuration unchanged. Check the result. If rate-limited again, recheck the attempt count before another resume.If using live services makes testing risky, costly, or hard to repeat, simulated services offer a practical alternative. They let you test the same scenarios across agents and tasks under controlled conditions. Just keep in mind that the results reflect the behavior you’ve simulated and issues you leave out may still surface when the agent uses the real service.
Provide the right context and permissions
Make the information the agent needs available where it'd encounter it in practice. This might include setup guides in your documentation, project instructions in the repository, or usage guidance in tool responses. Include a few tasks that require the agent to locate and use this information, so you can test both whether it finds the guidance and whether it applies it correctly.
Access is part of the task design, so decide whether the agent will start with an authenticated session, an API key, or a way to request access. Credentials let it connect to a service, while permissions determine what it can do once connected. Your success criteria should reflect both. If signing in requires a person, you might define success as reaching the login page, explaining what's needed, and asking the user to take over. This completes the handoff you're testing, even though the broader workflow still requires authentication.
We’re also watching auth.md, a protocol from WorkOS that helps agents discover how to register with a service and what access they can request. We’re excited to test how this works in practice.
Give each task a consistent starting point
An agent’s actions can change the environment for the next task. It might edit a file, update a record, or leave behind an authenticated session. If those changes carry over, the next agent faces a different task, making the results harder to compare. Reset the data, configuration, and any relevant history so each task begins under the intended conditions.
Run each task in its own isolated environment so its changes cannot affect other tasks or live systems. Reset the environment before the task begins to restore the intended data, configuration, and permissions. Together, these controls give each task a consistent starting point and keep its effects contained. This makes it easier to investigate unexpected results and compare performance as agents and products evolve.
| State | How carryover can affect the next task | What to restore |
|---|---|---|
| Files and repository | Previous edits or generated files may complete part of the task or introduce errors | The specified repository revision and initial file contents |
| Application data | Created, updated, or deleted records may change the task or expected result | The defined starting records and their values |
| Authentication | An active session or saved credential may let the agent skip an authentication step | The intended session and credential state |
| Permissions | Access granted during a previous task may allow actions that should require approval | The specified roles, access rights, and approval requirements |
| Configuration | Changed settings or environment variables may alter application or tool behavior | The defined application settings and environment variables |
| Background processes and pending work | Processes or queued jobs from an earlier task may continue changing the environment | The intended processes and pending jobs, with work from previous tasks stopped or cleared |
Conclusion
The evaluation environment shapes what you can test and what you can conclude from the results. Its tools, data, context, and permissions determine what the agent can attempt and what success demonstrates. Thoughtful choices here can reveal where your product makes work straightforward and where agents need better support.
You can learn a lot from a small, carefully designed test. Start with realistic tasks, reset the conditions between tasks, and verify the results. Repeat those tests as you improve your product to see whether the changes help agents complete the work more reliably.
FAQs
Common questions this article helps answer
