The perfect examination has two qualities: it measures the skill we care about, and the candidate cannot redecorate the marking scheme during lunch. AI evaluation is working on the second one.
Researchers at Berkeley examined vulnerabilities in a collection of agent benchmarks and demonstrated ways that an agent with access to its environment could undermine intended evaluation. Their examples included altering verification paths and exploiting exposed information. This was an audit of benchmark design, not proof that every high-scoring model routinely cheats.1
The machinery can produce a valid score for an invalid reason. The number survives; its advertised meaning does not.
The exam room is part of the exam
A benchmark is often presented as a number beside a model name. Behind that number is an environment: files, permissions, inputs, tools, checks, and assumptions about what the system is allowed to touch.
If the candidate can change the mechanism used to judge success, the benchmark may be measuring access rather than competence. An impressive result might reflect useful problem-solving, an unintended shortcut, or some combination. The number alone cannot separate them.
Imagine a hypothetical agent asked to repair a small application. The application should satisfy a set of tests. If the agent can alter both the application and the only record of whether the tests passed, success becomes ambiguous. We need a verifier whose independence does not depend on the candidate’s good manners.
Developers legitimately change faulty tests. An evaluation still needs an independent requirement against which to check the result. Otherwise, the agent can repair the exam instead of the application and receive the same congratulations.
We have optimized the scoreboard
It can also conceal the boundaries of the task. Did the system have the same tools? The same budget? The same opportunities to retry? Was the evaluation environment exposed during development? Were failures retained in the result? Which behavior actually produced the success signal?
These questions do not require assuming misconduct by a model maker. Evaluation is difficult even when everyone is trying to be careful. The more capable and flexible the evaluated system becomes, the more thoroughly it can explore assumptions the evaluator did not realize were assumptions.
The customer has a less editable test
In deployment, the organization usually wants an outcome outside the agent’s own report. The invoice must reconcile. The software must behave correctly for users. The appointment must exist in the right calendar. A reassuring summary does not supply these facts.
Here, too, independence matters. A completion message should not be the only evidence that the work happened. The system that requests an action, the system that permits it, and the system that verifies the result need not all have identical authority.
Verification should follow the stakes. Read an email draft before sending it; independently check a consequential action before relying on the agent’s completion message. The harder the action is to reverse, the less its author should control the only evidence that it worked.
A lower score might be better news
When an evaluation is hardened, reported performance may decline. In a marketing meeting, this looks unfortunate. In a measurement meeting, it can be a sign of progress.
The Berkeley audit puts the examination room under examination. That is overdue work in a market quite willing to discuss the candidate’s score without inspecting who had the keys.
Congratulations should come from a system the candidate was not allowed to edit.