A Benchmark Is a Game Against the Model

Gemini 3.8 Flash is either one of the best terminal agents in the world or not particularly good at terminal agents. Google’s own evaluation table supports both readings.

Model	Terminal-Bench 2.1	Terminal-Bench 4.0
Gemini 3.8 Flash	89.4%	19.1%
Gemini 3.7 Flash	85.8%	11.2%
Claude Opus 5	89.1%	51.8%
GPT-5.6 Sol	88.8%	37.3%

The same model that leads one version finishes more than thirty percentage points behind another model on the next. The obvious reaction is benchmark-maxing.

Here, benchmark-maxing means optimizing a model or agent harness for a public score in ways that may not transfer to a new task distribution. It is broader than cheating.

Perhaps Gemini was post-trained aggressively toward the older benchmark. Perhaps its agent harness was tuned around the tasks, instructions, and failure modes that Terminal-Bench 2.1 made visible.

That interpretation may be right, but the public evidence does not establish it.

The model card names the model, but it does not make the two benchmark runs a controlled comparison with an identical harness.

Terminal-Bench 4.0 is not merely a fresh sample from the same test. It changes task resources, fixes nineteen tasks, and removes eight—including tasks that had become saturated or had public solutions. The benchmark maintainers describe the release as a breaking change that requires new trials.

Gemini 3.8 also improves over Gemini 3.7 on the new version, from $11.2\%$ to $19.1\%$. The model did not simply become worse.

Something more interesting happened: the benchmark changed, and the meaning of the score changed with it. That small drama exposes a larger problem in AI evaluation:

A benchmark stops being a passive measurement device when performance on it changes what developers train.

At that point, the benchmark becomes a game against the model.

Benchmark exploitability is the public-score gain available without a corresponding improvement on hidden or future tasks. This extends the question behind my LLM policy audits: what does average performance hide about adaptation?

What Gemini 3.8’s Terminal-Bench scores show

It would be easy to treat $89.4\%$ and $19.1\%$ as contradictory estimates of one hidden quantity called terminal ability, but they are not.

A benchmark version defines a task distribution, an execution environment, an agent interface, a resource budget, and a scoring rule. Change enough of those components and the experiment measures a different object.

Terminal-Bench 4.0 gives every task an eight-hour agent timeout. It recalibrates CPU and memory, repairs misspecified verifiers, and removes tasks that no longer distinguish frontier systems. The maintainers report fewer infrastructure failures, leaving more failures attributable to model refusals or exhausted output budgets.

Those are substantive improvements, but they also make comparisons across versions difficult. That difficulty is precisely the point.

If a model’s apparent capability depends heavily on one public task distribution, then the leaderboard score was always local. We merely needed a new version to see the boundary.

The old score answered:

How well does this model-and-agent system perform on Terminal-Bench 2.1?

It did not necessarily answer:

How capable is this system at unfamiliar terminal work?

The second question is what most readers thought they were buying.

How a public benchmark enters post-training

The clean textbook picture has a model on one side and an independent test on the other. The laboratory trains the model, the benchmark measures it, and information flows in one direction.

Public benchmarks do not remain that clean. Once a leaderboard matters, it changes development decisions. Laboratories study failures, add related data, tune post-training rewards, modify system prompts, redesign tool interfaces, increase inference budgets, and build agent scaffolds around the benchmark’s demands.

Most of this is legitimate engineering. If a benchmark reveals that models cannot recover from a broken build, training them to recover is progress. If agents waste their context window reading irrelevant files, improving their search policy is progress.

The problem is not optimization itself, but how optimization changes the relationship between the benchmark and the capability it was intended to measure.

A public test becomes part of the training environment, even when its exact examples never enter the training corpus.

The same effect appears in conversational evaluation. AlpacaEval became known for favoring longer answers, so its maintainers introduced a length-controlled score. A response could gain preference by supplying more text, even when the extra length did not represent the improvement users cared about.

The evaluator rewarded a strategy, and models and developers found it. This is not an accident around the edges of evaluation; it is what optimization does.

A related problem appears in The Price of Seeing: when behavior changes which evidence becomes visible, passive measurement becomes misleading.

Four ways to improve an LLM benchmark score

Suppose a model’s benchmark score rises. At least four different things may have happened.

Swipe to inspect →

The underlying capability improved. The model became better at planning, coding, debugging, or tool use, and the gain transfers to unfamiliar tasks.

The model specialized to the benchmark distribution. It learned recurring task forms, common packages, instruction patterns, or verifier expectations that do not transfer as broadly.

The harness improved. Better prompts, tools, context management, retry logic, and timeout handling convert the same model into a stronger evaluated system.

The system exploited the measurement. It found public solutions, manipulated a verifier, or otherwise received reward without demonstrating the intended capability.

These mechanisms are not morally equivalent. Cheating violates the evaluation rules, reward hacking exploits a loophole, benchmark specialization can result from legitimate post-training, and harness engineering may produce real deployment value. Yet all four can raise the number printed in a model release.

Terminal-Bench has documented actual cheating and reward hacking on its leaderboards. In one class of failures, agents retrieved task solutions from the internet. In another, submitters accidentally or intentionally exposed privileged benchmark information.

There is no evidence in the published Gemini results that Google did either. The important point is structural: a leaderboard score alone cannot identify which mechanism produced it.

The model is not the only player

We can make the interaction more precise by representing a benchmark version as

$$B_t=(D_t,H_t,S_t),$$

where $D_t$ is the task distribution, $H_t$ is the evaluation harness and environment, and $S_t$ is the scoring rule.

The evaluated system is not only a set of model weights. It is

$$M=(\theta,A),$$

where $\theta$ is the model and $A$ is the agent scaffold: prompts, tools, memory, retries, and execution policy.

Once $B_t$ is public, a laboratory can choose $M$ to maximize

$$\mathbb{E}_{x\sim D_t}\left[S_t(M,x)\right].$$

The benchmark maintainer then observes failures, saturation, and reward hacks. The maintainer repairs verifiers, removes solved tasks, changes resources, and releases $B_{t+1}$.

The laboratory best-responds to the benchmark, and the benchmark maintainer best-responds to the laboratory. Evaluation has become a repeated game.

This gives a different interpretation of benchmark progress. A rising score may represent movement toward general capability. It may also represent movement toward equilibrium with one measurement system.

The two are identical only when the benchmark continues to predict performance outside itself.

Benchmark exploitability: a game-theoretic definition

In game theory, a policy is exploitable when an opponent can gain by adapting against it. A benchmark can fail in an analogous way.

Imagine an adaptation $a$ that changes the model, its post-training, or its harness. The adaptation produces a gain on public benchmark $B$, but we also evaluate it on a hidden transfer distribution $U$.

Define the benchmark-specific gain as

$$\operatorname{Exploit}(B;\mathcal{A},U) = \sup_{a\in\mathcal{A}} \left[ \Delta S_B(a)-\Delta U(a) \right].$$

Here, $\mathcal{A}$ is the class of adaptations available to the developer. Scores must be normalized before the subtraction is meaningful.

If an intervention improves the public benchmark and the hidden distribution equally, the difference is small. That looks like capability improvement.

If it produces a large public gain with little hidden transfer, the benchmark is exploitable by that intervention class.

This definition deliberately includes more than cheating. A model can exploit a benchmark without breaking a rule; it only needs to learn something the score rewards more than the world does.

The phrase benchmark exploitability therefore describes a property of the whole evaluation mechanism. A benchmark may resist prompt tricks but remain vulnerable to targeted post-training. It may resist data contamination but remain vulnerable to harness specialization.

There is no single exploitability number without specifying the opponent, just as there is no opponent-independent exploitability number in a game.

Every benchmark has a half-life

A useful benchmark begins with uncertainty. We do not know which systems can solve its tasks or what their dominant failure modes will be, so the first evaluations reveal something real.

Then the benchmark starts teaching. Public examples reveal the task distribution, agent traces reveal useful strategies, failed submissions expose verifier behavior, and successful systems suggest training data. Repeated evaluation concentrates engineering attention on whatever the benchmark rewards.

Eventually, score gains become harder to interpret. Call the duration of useful predictive power the benchmark’s half-life.

The half-life ends not when every task is solved, but when success on the benchmark stops carrying enough information about success on the next relevant task distribution.

AI may shorten this half-life dramatically because agents can inspect tasks, generate synthetic relatives, search for public artifacts, and run optimization loops faster than benchmark designers can author replacements.

A benchmark can remain technically difficult while becoming strategically familiar. That is a more dangerous form of saturation: the leaderboard still moves, but it may be measuring increasingly local adaptation.

Evaluate benchmark versions, not snapshots

The usual leaderboard compares many models on one benchmark version. We should also compare many frozen model-and-harness snapshots across many benchmark versions.

Imagine a version-transfer matrix. Each row is a frozen evaluated system. Each column is a benchmark version. Every cell contains a fresh run.

Swipe to inspect →

A broad capability improvement should remain useful on later columns, while a benchmark-specific intervention should create a spike near the version that shaped it.

The matrix would also help separate models from harnesses. Fix the model and vary the agent scaffold. Then fix the scaffold and vary the model. A score that changes mainly with the harness should be reported as an agent-system result, not a model property.

We could summarize the matrix through rank stability, normalized transfer, and task-family generalization.

The goal is not to punish specialization, because specialized systems can be enormously useful. The goal is to stop calling specialization generality merely because the benchmark name sounds broad.

Why continuous benchmarks should fight back

Terminal-Bench is moving toward a promising answer: treat benchmarks as maintained software.

Its continuous-benchmark proposal uses semantic versioning. Some changes permit old results to be reused. Verifier changes require regrading. Changes to tasks or agent environments require complete reruns.

This is more than convenient infrastructure because it gives the benchmark an explicit update policy.

A strong continuous benchmark can remove saturated tasks, repair accidental shortcuts, add failures observed in deployment, and test cheating agents against its verifiers. It can preserve reproducibility without pretending that a frozen test remains informative forever.

The benchmark thereby becomes an active participant in the evaluation game.

Static benchmarks are targets. Continuous benchmarks can become adversaries.

This does not mean trying to make every model fail. Good benchmarks should reward real progress while changing enough that the cheapest route to a higher score remains improvement in the capability we care about.

What should an LLM leaderboard row mean?

The Gemini 3.8 example is useful because the contradiction is visible. An $89.4\%$ score on Terminal-Bench 2.1 is real, and a $19.1\%$ score on Terminal-Bench 4.0 is also real. Neither number should be promoted into a timeless property of the model.

A serious leaderboard row should identify:

• the benchmark version;
• the model snapshot;
• the agent harness;
• the tool and network policy;
• the reasoning and token budget;
• the evaluation date;
• the uncertainty across trials;
• the task exposure and contamination policy.

Without these details, “model score” compresses an entire experimental system into one convenient label, which is often exactly what makes the result misleading.

Measurement becomes part of the system

Benchmarks have driven extraordinary progress in machine learning because they make disagreement concrete. They turn vague impressions into tasks, rules, and numbers, allowing laboratories to compare methods and discover failures that anecdotes would miss.

That success creates the strategic problem: when a benchmark matters, people optimize against it, so the benchmark influences the systems it measures. As those systems improve, they discover its shortcuts, blind spots, and obsolete assumptions.

The benchmark is now inside the loop. This does not make evaluation impossible; it means evaluation needs mechanism design.

We should ask not only whether a benchmark correlates with human judgment today, but how it behaves after laboratories spend millions of dollars learning to win it. We should also ask how rapidly its score transfers to future versions, which adaptations it rewards, and which opponent it was designed to withstand.

Every successful public benchmark eventually trains its own opponent, so the best benchmarks will be the ones that know how to play back.

Related reading

Research

LLM Exploitability Audits

Exact best-response audits reveal vulnerabilities that selected-opponent evaluations can miss.

↗Blog

The Price of Seeing

How an agent's actions change the evidence available for learning.

↗Blog

The Gap to AGI: Learning How to Think

Why general intelligence may require adapting the form of computation.

↗