When Models Become Examiners: Self-Play as an Answer to Benchmark Saturation

Introduction

Mathematical reasoning benchmarks are running out of headroom. MATH and GSM8K, once the standard tests for separating frontier language models, no longer discriminate between the strongest systems because nearly everyone scores near the ceiling. The community's reflexive response has been to manufacture harder problems: live competition sets, research-level questions, annually replenished pools. But this response treats benchmark saturation as a supply problem, as if the only obstacle were producing enough difficult questions.

A recent preprint, MathDuels: Evaluating LLMs as Problem Posers and Solvers by Xu, Jin, Arya, and Naik at the University of Pennsylvania, reframes the issue entirely. The bottleneck is not the quantity of hard problems. It is that humans, as problem authors, sit at a roughly fixed difficulty ceiling that advancing models climb past faster than we can raise it. The paper's proposed remedy is historically resonant and structurally elegant: let the models test each other.

The Self-Play Protocol

MathDuels is built on a sixteenth-century precedent. In 1535, the mathematicians Tartaglia and Fior deposited thirty problems each with a notary and competed to solve the other's set. Tartaglia solved all thirty of Fior's; Fior solved none of Tartaglia's. The duel exposed a capability gap that no shared static test could have surfaced, precisely because each competitor was forced to probe the boundary of the other's knowledge. Victory required not just solving skill but the ability to construct problems beyond the opponent's reach.

The paper operationalizes this into a four-phase protocol over a set of N models, each authoring K problems:

• Generation. Each model authors problems through a three-stage pipeline: meta-prompting, problem generation, and adversarial difficulty amplification. The amplification step explicitly pushes authors to produce problems designed to defeat strong solvers.
• Solving. Every model attempts every problem it did not author. The outcome is a binary correctness matrix where entry y_mp records whether solver m solved problem p, judged automatically through symbolic verification.
• Verification. Any problem that defeats at least one solver undergoes a validity check to filter ill-posed or ambiguous questions. This is a crucial safeguard, since an unconstrained adversarial author could simply generate unsolvable nonsense.
• Ranking. A Rasch model jointly estimates solver abilities s_m and problem difficulties d_p by maximizing the log-likelihood of the outcome matrix, with an L2 penalty on difficulties for regularization. Author quality is then derived from the mean difficulty of each model's authored problems, and a composite rating blends solve and author ratings via weights w_solve and w_author.

The Rasch model deserves a note. Borrowed from psychometrics and item response theory, it models the probability that solver m answers problem p correctly as a logistic function of the difference between solver ability and problem difficulty, σ(s_m - d_p). This is the same family of latent-trait models that underlies standardized educational testing. Using it here is principled: it disentangles whether a problem went unsolved because it was genuinely hard or because the solver attempting it was weak. A naive accuracy count cannot make that distinction; the Rasch formulation can, because difficulty and ability are estimated jointly from the full matrix of confrontations.

The Decoupling Result

The headline empirical finding, drawn from experiments across 19 frontier models, is that authoring capability and solving capability are partially decoupled. Strong solvers are not necessarily strong authors. The reported solver-rating versus author-rating scatter plot shows models scattered off the diagonal rather than clustered along it. A model can construct problems that defeat solvers it cannot itself outperform on solving.

This is more interesting than it might first appear, and worth dwelling on. In a single-role benchmark, a model is collapsed onto a single scalar: its accuracy on a fixed problem set. MathDuels argues that mathematical competence has at least two partially independent axes, and that the conventional benchmark conflates them or ignores authoring entirely. The act of constructing a hard, well-posed, verifiable problem is a distinct skill from solving one. It requires anticipating where a solver will stumble, controlling the answer so it remains checkable, and avoiding the trap of producing something merely ambiguous rather than genuinely difficult.

The second finding follows from the design rather than from any single measurement. As stronger models enter the arena, they author problems that defeat previously dominant solvers. The benchmark's effective difficulty co-evolves with participant strength instead of saturating at a fixed ceiling. This is the structural payoff of self-play: the difficulty frontier is endogenous to the population of competitors, not exogenously fixed by human authors.

My Take

The decoupling result is the part of this paper I find genuinely consequential, and I want to push on what it implies. If authoring and solving are partially independent axes, then "mathematical capability" as a single number is a lossy compression. We have been ranking models on solving alone because that is what static benchmarks measure, but the ability to generate problems at the boundary of another agent's competence is arguably closer to what we mean by mathematical creativity. Solving is search within a known problem space. Authoring a discriminative, well-posed problem is closer to constructing the space itself.

There is a connection here to a broader theme in machine learning: self-play as a route past the limits of human-supplied supervision. AlphaGo Zero discarded human games and improved through self-play; the curriculum emerged from the agents themselves. MathDuels imports that logic into language model evaluation. The difference, and it is an important one, is that Go has a ground-truth reward (you win or you lose), while mathematical correctness for novel adversarial problems must be established through the verification step. The entire protocol's validity rests on that verifier. If the verification phase admits subtly ill-posed problems, or rejects hard-but-valid ones, the difficulty signal is corrupted. The paper's symbolic verification and validity check are the load-bearing components, and I would want to see their false-positive and false-negative rates characterized carefully before trusting the leaderboard at the margins.

A second concern is the adversarial incentive itself. A model rewarded for authoring unsolvable problems has an incentive to drift toward the boundary of well-posedness, where ambiguity masquerades as difficulty. The verifier is the only thing standing between "hard" and "broken." This is a delicate equilibrium, and I suspect it is the place where the protocol is most fragile. It also raises a subtler question: are the hardest authored problems hard because they are deep, or because they are computationally tedious, or because they exploit a known weakness in solver parsing? The Rasch difficulty parameter does not distinguish these.

There is also a self-reference worth flagging. When the population of authors and the population of solvers are the same models, the difficulty ceiling co-evolves but it co-evolves within the envelope of what current models can conceive. A blind spot shared across all participants, a class of problems no model thinks to pose, remains invisible. Self-play raises the ceiling, but only along directions the population already explores. This is a real limit on the claim that the benchmark "resists saturation by design." It resists saturation in difficulty, not necessarily in coverage.

Conclusion

MathDuels makes a reframing that I think is correct and overdue: benchmark saturation is not primarily a data-supply problem, it is a consequence of pinning problem difficulty to a fixed human ceiling. By casting models in dual roles and letting them probe each other's boundaries, the paper offers a benchmark whose difficulty scales with the field it measures. The decoupling of authoring and solving is the kind of result that should change how we talk about model capability, since it shows that the scalar rankings we rely on are flattening a multidimensional skill.

The open questions are about robustness. How reliable is the verifier under adversarial pressure? Can the difficulty signal be gamed by tedium rather than depth? And what happens to coverage when the authors and solvers share blind spots? Self-play in mathematics is a promising direction, but its credibility lives or dies on the integrity of the judging loop. If that holds, the live leaderboard at mathduels.ai may turn out to be one of the more durable evaluation instruments we have, precisely because it refuses to stand still.