The work developers ask agents to do with your product gives you a great starting point for evaluating how well those agents perform. Those everyday requests are the beginnings of a useful evaluation dataset. Turn them into tasks, include the variations developers are likely to encounter, and decide what information the agent should have and how you’ll verify the outcome. Over time, the results can show where your product supports agents well and where an improvement could make the next developer’s experience smoother.
Create a task dataset for AI agent evaluation
A task dataset gives you a structured way to evaluate how agents handle everyday developer workflows with your product, from installing a CLI to troubleshooting an error. Each task sets out what the agent needs to accomplish, the data and conditions it starts with, and how you’ll check whether it succeeded.
The five steps below explain how to build evaluation tasks, from choosing workflows to checking task quality and keeping your dataset current.
Build tasks around real developer workflows
Build your first dataset around the work users depend on, from getting started to completing core operations and resolving common problems. Draw from:
- Documentation
- Support requests
- Manual checks
- Reported failures
Prioritize tasks by how often users encounter them and the consequences of getting them wrong. Group tasks by workflow to spot gaps in coverage and track where product improvements help.
Give each task a clear goal and success criteria
Give each task a clear goal and identify the outputs or changes you’ll check to confirm success. Record the starting conditions, make the requirements clear to the agent, and allow different approaches that satisfy them. Match the guidance to what you want to evaluate, leaving room for agents to find the relevant documentation when discovery is part of the task.
Test how agents adapt as conditions change
Build on a working task by introducing everyday complications, such as:
- An expired credential
- An unspecified destination
- A resource that already exists
Define the expected outcome for each variation, including when success means asking for clarification, requesting approval, or explaining why the task cannot proceed. This expands the evaluation to cover the decisions that make the workflow useful in practice.
Pilot tasks and verify the checks
Give each task a practice run to confirm that it’s clear and achievable.
- Test the checks against known valid and invalid results. They should accept different valid solutions and reject work that falls short of the requirements.
- Review passing and failing transcripts in Quantiles to see whether problems come from the agent’s work or the evaluation itself.
Once the task and checks are ready, run several trials to assess how consistently agents complete it. Reset the environment to the task’s defined starting conditions before each trial. Quantiles automatically runs multiple trials per task, with no extra work.
Add and update evaluation tasks regularly
Keep your dataset connected to how people use your product as their needs and your features evolve.
- Add cases that expand coverage.
- Retain reliable workflows as regression checks.
- Track changes to task instructions, starting data, and validation.
Keep some tasks out of the process you use to refine documentation or skills, then use them to check whether improvements carry over to other workflows.
Conclusion
A useful dataset can begin with a few well-chosen tasks that reflect real work and have outcomes you can verify. Let the results guide what you improve and which tasks you add next. As your product evolves, those tasks give your team a way to revisit the experience your users depend on and see whether agents are finding it easier to get things done.
FAQs
Common questions this article helps answer
