Evaluating a machine negotiator before procurement

Seven statements, each about the models under the card above; the mapping to procurement is a reconstruction R, and no statement is an assessment of any agent named or described in item 1.

• A pilot result is a head-to-head average. An agreement rate, or a win rate against the counterparties who happened to take part, is Model W1's first number. In that model a fixed doctrine earns $+2$ per round against one sparring partner while guaranteeing $-3$ in a game of value $0$ (Proposition 12.2 (p. 491)(a)). The object that survives a change of counterparty is the guarantee, and the distance from the best attainable is the exploitability. A reported agreement rate is one step further from either: it counts deals, not deals on good terms D [Proposition 12.2 (p. 491); the reading of agreement rates is R].

• Evenly matched agents show $70\%$ or more. Under Model W2 both players are exactly optimal, and conditional on a decisive round each wins with probability exactly $\frac{1}{2}$. An agent of that kind wins at least $7$ of $10$ evaluation negotiations with probability exactly $\frac{11}{64}=0.171875$, and at least $14$ of $20$ with probability $0.0576\dots$ (Proposition 12.10 (p. 497)(a), Table 12.2 (p. 486)a). A vendor's “$70\%$ in ten negotiations” is therefore a result that an agent no better than its counterparties produces about one time in six D [Proposition 12.10 (p. 497)].

• The evaluation needs tens of negotiations, and the number is not monotone. To separate a win probability of $\frac{7}{10}$ from $\frac{1}{2}$ with size at most $\frac{1}{20}$ and power at least $\frac{4}{5}$, a threshold rule needs at least $37$ negotiations (declare the agent better at $24$ wins or more); with power $\frac{9}{10}$, $53$; with size $\frac{1}{100}$ and power $\frac{9}{10}$, $80$. Because the binomial is discrete, a larger evaluation can be worse: at $38$, $39$ and $41$ negotiations no threshold rule meets the first pair of targets, and only from $42$ does every larger $n$ checked do so (Proposition 12.10 (p. 497)(b), Table 12.2 (p. 486)b). A protocol that says “at least $n$ negotiations” has to check the $n$ it actually runs D [Proposition 12.10 (p. 497)].

• Selection is the reporter's free parameter. If ten configurations of an agent (prompts, versions, settings), each with true win probability $\frac{1}{2}$, are piloted ten times each and the best is reported, its expected win rate is $0.7381\dots$, above $\frac{7}{10}$; the best of five shows at least $7$ of $10$ with probability $0.6105\dots$ (Proposition 12.10 (p. 497)(c), Figure 12.4 (p. 485)a). Applied to the best of five, the $37$-negotiation rule of item (iii) has size $0.2239\dots$ instead of $0.0494\dots$ This is Model W2's best-of-five arithmetic in a binary record, and it does not require bad faith: tuning an agent on pilot results and then reporting those results is selection. The remedy is a commitment in field 4 — counterparties, number of negotiations and acceptance rule fixed before the data, and evaluation data the vendor has not seen. The memorandum's request that independent-evaluation data not be accessible to the vendor is, in this model, exactly the device that removes the selection margin D [Proposition 12.10 (p. 497); the reading of the memorandum is R].

• A red-team result is one-sided. A deviation that is found and beats the agent proves that the agent's exploitability is at least the gain found, under the protocol of the test; a search that finds nothing proves only that weaknesses are rare in what was searched. Model W1's learning tester reports $-2$ at budget $1$ against an agent whose true exploitability is $3$ (Proposition 12.2 (p. 491)(b)). A blind search does no better: if one deviation in a hundred exploits the agent, $299$ independent probes are needed to find one with probability $\frac{19}{20}$, and a hundred probes find nothing with probability $0.3660\dots$ (Proposition 12.10 (p. 497)(d), Figure 12.4 (p. 485)b). The published finding that a scripted show of desperation raised an agent's payoff against a standard model is a report of the first kind: a lower bound on that model's exploitability in that scenario, not a measurement of it D [prop:sim-score,prop:sim-eval; Bianchi et al. 2024; the reading is R].

• Upsets are what pool dependence predicts. The finding that the strongest models sometimes lose to weaker opponents is, in Model W1's terms, the gap between head-to-head results and quality: the doctrine “always P” beats a partner who always plays R, and the wrong-model agent ties the equilibrium exactly in friendly play while being exploitable for $\frac{2}{7}$ (Proposition 12.2 (p. 491)(a)). A cross-play table ranks agents against the pool that played; a competition with more than a hundred thousand negotiations reduces the noise in that ranking without changing what it ranks against D [Proposition 12.2 (p. 491); Davidson et al. 2024; Vaccaro et al. 2025; the reading is R].

• Reproducible is not the same as replayed. A demonstration that replays fixed scenarios against scripted counterparties scores a transcript, and by Remark 12.3 (p. 492) a transcript can be exploitable well beyond the mixture it realises. An evaluation can be reproducible — as the memorandum asks of vendor-run tests — and still randomise genuinely, if its random choices are seeded and declared; what it must not do is score the replay of a sequence the agent was tuned on R.

Negotiation is not zero-sum and has no certified value: Model W1's exact column has no field counterpart, and “win” has to be defined against a baseline — usually what a human negotiator would have obtained from the same counterparty — which makes the evaluation a paired comparison, with its own variance, rather than a count. Outcomes are multi-dimensional (price, payment terms, delivery, the relationship), so a binary record discards most of what the buyer cares about. Negotiations are not independent: the same counterparties recur, learn the agent's habits and adapt, so the deployment population drifts away from the evaluation population; this is the NIST framework's caution about measurement in the laboratory, met in this setting. The agent is not fixed: a language-model agent is updated by its vendor, and a version evaluated before award is not the version deployed a year later. And the alternative $\frac{7}{10}$ is arbitrary; the difference worth detecting is set by the value of a better agent to the buyer, which the model does not contain.