When the agent says it worked

The claim and the check

Every agent that finishes a task produces two things: a change in the world, and an account of that change. The account is immediate, legible, and free. The change is slow, scattered across systems, and usually owned by somebody else.

So the account is what gets read. The agent says the ticket is resolved, the refund is issued, the access is restored. Somebody, or something, reads that and marks the run good.

False success is the gap between the two: the agent asserts completion when the state of the environment says otherwise. It is not a hallucination in the usual sense. The reasoning is sound, the summary is clean, the tone is confident. Only the world disagrees.

Why more checks do not help

The instinct when an agent reports success it did not achieve is to add checks. More evaluation steps, a stricter rubric, a second model reviewing the first. It rarely works, and the reason is structural rather than a matter of effort.

Almost everything available to check with comes from the agent itself. Its logs, its intermediate reasoning, its summary, its self-reported status. If the agent is wrong about having finished, every check downstream of it inherits that error. The rubric gets applied to a faithful account of a run that did not happen.

This is not a bug that a better prompt closes. It is a missing category of evidence. Any check built from the run's own output is testing internal consistency, which a confident agent will pass, and which tells you nothing about whether the world changed.

The study below is useful precisely because it breaks that circle. Its labels on one of the two benchmarks come from a database write and a programmatic test, with no dependence on the transcript. That is what makes it possible to ask, for the first time with a straight face, whether reading the transcript works at all.

What the study measured

Advani (2026)¹ examined 9,876 agent trajectories from tau2-bench across eight model families, and 1,879 from AppWorld across four. The AppWorld corpus matters disproportionately, because there the ground truth is fully independent of anything the agent wrote: the agent records a status field in a supervisor database, and unit tests check whether the environment actually changed. The two never touch.

How often agents assert success the environment does not support varied sharply by setting:

• 45 to 48% of failures in single-control tau2-bench domains, where the agent is the only party writing to the environment.
• 75.8% of failures among AppWorld coding agents that report their own completion status. The paper is explicit that this covers a subset of that benchmark rather than all of it.
• 3% of failures in one dual-control domain. That one is discussed below.

Why the judges failed

The obvious defence is to have a second model read the transcript and assess it. The study tested that directly, and generously: five judge models, five prompting strategies, and in some conditions the full task specification, which is more context than the agent itself had.

No configuration exceeded 0.65 AUROC on tau2-bench. On AppWorld API-call traces, the ceiling was 0.54. For orientation, 0.50 is what you get by guessing.

The mechanism is the part worth carrying. In the paper's words, judges rely on “surface completion proxies — confident closing language in tau2-bench and coarse action-sequence volume in AppWorld — rather than verified state changes.”¹ A run that ends with a tidy summary reads as successful, because a tidy summary is what success has always looked like in text. The judge is not short of information. It is anchoring on the wrong part of it.

The number that matters operationally

AUROC is a ranking statistic and a poor guide to what a team would actually experience. The study also reports deployment metrics, which are more useful. Assume you can afford to review one run in ten:

At the same budget, a bag-of-words classifier recovers more than five times what the strongest judge does, and runs roughly 3,300 times faster. The paper is equally clear about the cost: “The detector is a triage signal, not an autonomous monitor. At a 10% flag rate, precision is 50%: half of flagged trajectories are false alarms.”¹

The domain where it nearly vanished

One tau2-bench domain recorded 3% rather than 45 to 48%. It is the only dual-control domain in the study, meaning an independent process could verify what the agent had done.

This is the most suggestive result in the paper for anyone building verification, and it is also the one the author hedges hardest. There is a single dual-control domain available, with fifteen false-success cases, which the paper says is insufficient to separate environment structure from other differences between domains. It is offered as an observation, not a cause.²

We take it the same way: a hypothesis worth testing, not a proven mechanism. It happens to be the hypothesis this company was built on, which is a reason to be more careful with it, not less.

What the paper does not say

Four limits, stated plainly, because the argument is weaker if they are buried.

It does not say the signal is absent from the transcript. The winning method reads the transcript. It is a TF-IDF classifier over trajectory text, reaching 0.83 and 0.95 task-disjoint AUROC. What fails is asking a language model to read the transcript and judge it. Those are different claims and only the second one is supported.

It does not generalise to production. Both corpora are simulations: a customer-service environment with a scripted user simulator, and a personal-app sandbox. The paper states that whether this extends to unstructured settings such as web navigation or open-ended code generation remains open, and that the detector “has not been tested in production settings.”¹

It is not a large peer-reviewed result. It is a single-author paper accepted to the Failure Modes in Agentic AI workshop at ICML 2026. Workshop review is real but light, and no independent replication exists at the time of writing.

It does not recommend what we sell, as its primary recommendation. Its headline advice is a cheap calibrated detector used for triage.

What follows from it

The paper's own conclusion draws the line we would draw, and it is worth quoting exactly: “production monitoring should treat lightweight, domain-calibrated detectors as triage signals for false success, while reserving direct trajectory–environment consistency checks for higher-stakes deployment.”¹

It says the same thing twice more. In the discussion: “Substantial gains would likely require methods that verify trajectory-environment consistency directly rather than reading surface text.” In the limitations: “High-stakes deployment likely requires direct trajectory-environment consistency checks rather than surface-text detection alone.”¹

So the ordering is: cheap text-based triage where volume is high and stakes are low, and direct comparison against the environment where the stakes are real.

Underneath both sits a structural point that survives every caveat above. Every method in the study that worked had to be calibrated against ground truth, and that ground truth came from the environment. A detector cannot be trained without it. The judges could only be shown to be failing because something outside the transcript said what had actually happened. Whatever you build, the transcript cannot be both the evidence and the verdict.

Which returns to where this started. The problem was never that the reader was not clever enough. It was that there was only ever one source, and it belonged to the party being assessed. What the winning method had was not more intelligence. It was a second source: records of what happened afterwards, used to calibrate what to look for.

For anyone running agents against real outcomes, that second source already exists and is already yours. It arrives later than the run, in a different system, usually owned by a different team. The ticket that reopens. The payment that never settles. The change reverted on Tuesday. Nobody joins it back to the run that caused it, which is the work, and by this study's own account it is also what stops the failure mode at its source.