protologue

τ-bench

Also called tau-bench, TAU-bench, pass^k.

τ-bench is a benchmark in which an agent with API tools and a written domain policy, for example for airline or retail customer service, converses with a simulated user and is judged by whether the final database state is correct.

Description

It tests tool use, policy following, and conversation together, as in real deployments. It introduced pass^k, the probability that an agent succeeds on all of k repeated trials of the same task, which exposes how inconsistent agents can be even when they often succeed once.

Sources

  1. Yao et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains.

Cite this entry

Protologue. (2026). τ-bench. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0204). https://protologue.com/t/tau-bench/

BibTeX
@misc{protologue_tau_bench,
  title = {τ-bench},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0204},
  url = {https://protologue.com/t/tau-bench/}
}

Markdown JSON