READ THE SUCCESS CONDITION · BY ANTI HUNTER
ONCE OR
EVERY TIME?
I want the word after “success” before I want the percentage.
HumanEval’s pass@k asks whether at least one of k independently sampled attempts passes the evaluator. Tau-bench’s pass^k asks whether every one does. One measures finding a successful attempt; the other measures repeatability under that task’s trial conditions.
STIPULATED FIXED POOL · NO MODEL OUTPUTS
Five attempts. Ten pairs.
Assume A, B and C pass; D and E fail. Choose two distinct attempts from this saved pool. Order does not matter.
- A · PASS
- B · PASS
- C · PASS
- D · FAIL
- E · FAIL
| Pair | At least one passes? | Both pass? |
|---|---|---|
| A + B | Yes | Yes |
| A + C | Yes | Yes |
| A + D | Yes | No |
| A + E | Yes | No |
| B + C | Yes | Yes |
| B + D | Yes | No |
| B + E | Yes | No |
| C + D | Yes | No |
| C + E | Yes | No |
| D + E | No | No |
| Pairs meeting the condition | 9 of 10 90% | 3 of 10 30% |
Same attempts. Different question. These are exact subset proportions for this fixed pool, not known probabilities for future runs.
A sample is not a known success rate.
With n attempts and c passes for a task, the papers use 1 − C(n−c,k) / C(n,k) for at least one and C(c,k) / C(n,k) for every one, then average across tasks. Here C counts distinct subsets, n ≥ k ≥ 1, and C(a,k) is zero when a < k.
The fixed-pool fractions above illustrate those estimators. Their unbiasedness concerns averages over repeated independently sampled pools. It does not make 90% or 30% a known probability for this task’s next runs.
For a known per-task success probability p and independent identical trials, the corresponding probabilities are 1 − (1−p)k and pk. Plugging an observed overall pass rate into these powers changes the calculation. Task averaging and these nonlinear operations do not generally commute.
Finding success still leaves a selection problem.
An evaluator can recognize that one candidate passed without giving a deployed system a reliable way to choose it. Selection, verification and retries still need a plan. That is an operating inference, not a benchmark result.
Before comparing percentages, check the task, evaluator, attempt settings, k, and whether success means at least one or every one. Whole-task reruns are different from consecutive steps in a workflow.
Sources and limits.
Evaluating Large Language Models Trained on Code, §2.1 ↗ and Appendix A ↗ (2021, version 2); tau-bench, §3 ↗ (June 2024, version 1). Methods read on .
This explains historical metric definitions, not current model rankings or production reliability. The examples are stipulated. No model was run. Tau-bench uses simulated users and notes that its reward can miss a policy violation; an evaluator pass is not universal correctness.