ZeroSum-Bench

The game

Models negotiate directly against each other for money. Every model plays every other model in ordered pairings (12 models, all 144 pairings, on the order of a thousand episodes per model), so a score is earned against the whole field, never against a fixed scripted opponent. What one side gains, the other side gave up: the benchmark measures who walks away with the money that actually changed hands.

Three mechanisms

Each pairing meets across three classic economic mechanisms: bilateral trade (a buyer and a seller haggling over a deal with private values), a first-price auction (bids are committed, the winner pays their bid), and a provision-point game (a shared pot that pays out only if contributions clear a threshold). The mix rewards different skills: price discovery, bid shading, and credible commitment.

What the numbers mean

Claim share is the fraction of the realized surplus an agent keeps for itself; 50% is an even split, and higher means it takes more from the agents it faces. Efficiency is how much of the available surplus a pairing realizes at all: an auction that overshoots destroys money nobody gets. The dominated rate tracks how often an agent plays a move that is strictly worse than an available alternative, regardless of what the opponent does.

The headline metric is money extracted as a share of what changed hands, averaged across the three mechanisms and the full field of opponents. The head-to-head view breaks the same episodes down by pairing: edge is how many points of the available surplus one model takes above or below a specific opponent, so a model can look strong on average and still have a losing matchup.

The benchmark design builds on the strategic-negotiation setting in the paper linked below the leaderboard; the arena, the pairing schedule, and the scoring are ours.