τ-bench

Overview

τ-bench evaluates language agents in dynamic customer-service episodes rather than single-turn tool-use prompts. Each episode gives the agent access to domain policy text and API tools, while a language model simulates the user. The agent must gather missing information, follow domain restrictions, choose valid tool calls, and communicate the outcome to the user.

The original benchmark introduces two domains: τ-retail and τ-airline. These domains test realistic support workflows such as returns, exchanges, refunds, reservation changes, cancellations, and rebooking. The agent cannot see the hidden task annotation or raw database state. It can only inspect or mutate state through the provided APIs.

The central design choice is final-state evaluation. Instead of requiring one exact conversation or one exact sequence of read-only calls, τ-bench scores whether the resulting database state matches the task’s annotated target state and whether required information appears in the agent’s responses.

Dataset Specification

Size

• τ-retail contains 500 users, 50 products, 1,000 orders, 115 tasks, and 15 API tools (7 write and 8 non-write tools.).
• τ-airline contains 500 users, 300 flights, 2,000 reservations, 50 tasks, and 13 API tools (6 write and 7 non-write tools.).

Source

The benchmark was constructed from manually designed database schemas, API tools, and domain-specific policies, followed by LM-assisted generation of synthetic database entries and manual task scenario generation, goal-state annotation, and verification.

Input Format

A τ-bench task is a multi-turn environment interaction, not a single prompt-response example. Its configuration typically includes:

• domain: the benchmark domain, such as retail or airline for the original τ-bench.
• task_id: the customer-service scenario being evaluated.
• user_scenario: hidden instructions sent to the user simulator, including the user’s reason for calling, known information, unknown information, goals, preferences, and constraints.
• policy: domain-specific rules and procedures visible to the agent.
• tools: read and write API tools available to the agent for inspecting or mutating domain state.
• database: the hidden initial domain database state, which the agent can access only through the provided API tools.
• evaluation_criteria: hidden scoring criteria, including a reference action trajectory used to derive the target database state, required communication outputs, and the reward basis.

Task example:

{
  "domain": "retail",
  "task_id": 19,
  "user_scenario": {
    "visibility": "hidden_from_agent",
    "summary": "Mei Davis wants to return a water bottle and compare the savings from exchanging a pet bed and office chair for the cheapest available versions."
  },
   "available_to_agent": {
      "policy": "Retail support policy text",
      "tools": [
        "find_user_id_by_name_zip",
        "get_user_details",
        "get_order_details",
        "get_product_details",
        "return_delivered_order_items"
      ]
    },
  "hidden_environment_state": {
    "database": "Retail users, orders, products, payment methods, and order-item state loaded by the benchmark environment."
  },
  "evaluation_criteria": {
    "reference_actions": [
    {
      "name": "find_user_id_by_name_zip",
      "arguments": {
        "first_name": "Mei",
        "last_name": "Davis",
        "zip": "80217"
        }
      },
      {
        "name": "get_user_details",
        "arguments": {
          "user_id": "mei_davis_8935"
        }
      },
      ...
}

Output Format

A run produces a full trajectory and a final scoring record. The conversation and tool-call path can vary across trials, but successful runs should produce the target database state and communicate any required information to the user.

• messages: user, agent, and environment/tool messages from the simulated episode.
• tool_calls: API tool calls issued by the agent, including read-only calls and state-changing calls.
• db_check: whether the resulting database state matches the target database state derived from the task annotation.
• communicate_checks: whether required user-facing information appears in the agent’s responses.
• reward: binary episode reward, typically computed from database-state correctness and required communication checks.

{
 "domain": "retail",
 "task_id": "retail:<source-task-index>",
 "trial": 0,
 "trajectory": {
    "messages": [
      {
        "role": "user_simulator",
        "content": "I want to return my water bottle and see how much I would save by exchanging the pet bed and office chair for the cheapest versions."
      },
      {
        "role": "agent",
        "content": "I can help with that. I’ll look up your account and orders first."
      },
      {
        "role": "agent",
        "content": "The water bottle return has been processed. Returning it saves $54.04. Exchanging the pet bed and office chair for the cheapest versions would save $41.64."
      }
    ],
    "tool_calls": [
      {
        "name": "find_user_id_by_name_zip",
        "arguments": {
          "first_name": "Mei",
          "last_name": "Davis",
          "zip": "80217"
        }
      },
      ...
}

Metrics

• Episode reward: binary reward computed as r_action * r_output. r_action checks whether the final database state matches the unique ground-truth outcome database. r_output checks whether required information appears in the agent's response.
• pass^1: the average episode reward across tasks. This is the the main single-trial comparison metric.
• pass^k: reliability metric that estimates the probability that all k independent trials of the same task succeed, averaged across tasks. τ-bench emphasizes pass^k because customer-service agents need consistent task success, not occasional success.
• Diagnostics: task-level failures, tool-call traces, user-agent messages, turns, and runtime metadata are useful for debugging, but are not substitutes for final reward or pass^k.

Known Limitations

• The domains are simplified versions of retail and airline operations. They include realistic constraints, but they do not cover the full complexity of production customer-support systems.
• Rule-based final-state scoring can miss policy violations that still arrive at the correct database state, such as acting before explicit user confirmation.
• LM-simulated users add useful stochasticity, but they may not match the full range of real human behavior, ambiguity, emotion, escalation, or adversarial interaction.
• Results depend on the agent strategy, user-simulator model, sampling settings, maximum turns, API behavior, and implementation details such as concurrency or retry handling.
• Because τ-bench is public, tasks and trajectories may appear in model training or post-training data. Treat contamination as a risk when interpreting scores.
• The repository README notes that later repositories include task fixes and newer domains. If you need to compare against the 2024 paper, pin the original implementation and task set.

Versioning and Provenance

Record the benchmark version, task split, domain, task IDs, policy files, tool definitions, database fixtures, scoring implementation, reward basis, agent strategy, agent model and provider, user-simulator strategy, user model and provider, prompt templates, sampling parameters, maximum turns, number of trials, seed, runtime/package versions, concurrency settings, relevant environment configuration, and any local patches.

These details can affect episode reward, pass^k, cost, latency, reproducibility, and comparability across runs.

References

Yao et al., 2024. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.

Paper: https://arxiv.org/abs/2406.12045

GitHub Repository: sierra-research/τ-bench

Related Benchmarks