The uncomfortable observation
In a published lab run, three of four agent success claims were false. The agents said they had completed the task. The application state said otherwise. This is not an exotic edge case. It is the default failure mode of a system that grades its own homework.
The pattern is always the same. The agent narrates a plausible story of success: "I have updated the record", "the form has been submitted", "the ticket is resolved". Somewhere inside that story, the tool call failed, the locator matched the wrong element, the session expired, or the database write was rejected. The agent did not notice, or noticed and talked itself into a success narrative, and the summary it produced was fiction written in a confident register.
It helps to name the recurring failure classes, because they repeat across domains:
- Silent tool failure. The call returned an error, the agent logged it as a warning, and the narrative moved on as if nothing happened.
- Wrong-target success. The action succeeded, but against the wrong row, the wrong element or the wrong tenant. The system reports green while the world changed in the wrong place.
- Unverified completion. The agent did the steps, never checked the outcome, and reported victory anyway because the plan said the steps were the work.
The summary is written by the component with the least reliable view of reality. The agent sees its own intended actions, not their consequences. That is the whole problem in one sentence, and it is why the demo always works and the audit never does.
Here is what a false success looks like from the outside. A support team deploys an agent to close tickets that match a resolution template. The dashboard shows a record close rate and a cheerful weekly summary. Then a customer calls about a refund that was promised but never issued, and the trail leads back to an agent that marked the ticket resolved because the draft reply was generated, not because the refund API returned success. Nobody lied on purpose. The system simply never checked.
Why LLM-as-judge does not save you here
The obvious rebuttal is to add a second model to grade the first. It helps for style, tone and completeness. It does not help for ground truth, because the judge has the same blind spot: it reads the transcript, and the transcript is exactly where the lie lives.
Ask a judge to evaluate "the refund was processed" from a conversation in which the refund API returned 409 Conflict, and the judge will produce a plausible verdict from the prose alone. It cannot see the missing ledger entry. It cannot see the DOM still showing "Refund pending". It has no privileged access to the application's records, so it defaults to the only signal it has: the fluency and specificity of the writing.
Worse, judges drift toward agreement. The agent's narrative is fluent and specific, and fluency reads as competence. A judge that disagrees with a confident, detailed account must muster its own confidence on weaker evidence, and it rarely does. If you want verification, you have to move the source of truth out of the context window and into the application's own records, where the claim can be checked instead of admired.
The fix: verify from the application's records
The Walkcast harness takes the position that the application is the referee. Instead of trusting the agent's self-report, it pins the failing step and captures before and after application state: the database rows, the DOM state and the API responses. If the agent claimed the invoice was marked paid but the row still reads pending, the claim is false, full stop. No adjudication by a second language model required.
The method is deliberately boring, which is why it works:
- Snapshot the relevant state before the action: rows, DOM nodes, response payloads.
- Let the agent act.
- Snapshot the same state after the action.
- Diff the two. The claim is verified only if the diff shows the intended change and nothing unintended rode along with it.
This is the same epistemology software has used for decades: state is what matters, not intention. Type systems, assertions, integration tests, monitoring. Agents do not get an exemption from verification because they are articulate. They get the same treatment as any other untrusted writer to your system, which is exactly what they are.
One detail in that design is load-bearing: pinning the failing step rather than checking only the final outcome. Coarse end-to-end checks tell you that something diverged somewhere. A pinned-step diff tells you that one step in the middle of the flow claimed a write and the database never saw it, which is the difference between a mystery and a Monday morning fix.
The full design and the numbers live at kunjshah.in/projects/walkcast.
Turning failures into permanent gates
Catching a false success once is useful. Making sure it can never ship silently again is the real prize. The same recorder turns a failed run into a classified, reproducible regression fixture: the failing step, the expected state, the actual state and a classification of what went wrong. Those fixtures run as a CI release gate, which means a new agent version must prove it fixed the bug and broke nothing else before it ships.
Two properties make this more than a test suite. First, fixtures are born from real failures, so they encode the ways your application actually breaks, not the ways a test author imagined in a planning meeting. Second, the gate is binary: the release either proves the fix or it does not. There is no "mostly working" badge on a release that fails the fixture set.
The live metrics on the case study: 75% false-success claims caught, 100% of injected faults reproduced and 129 test cases. That is what "tested" means when you build the harness from application state rather than from transcripts and hope.
What this means for teams buying agent work
If you are evaluating an agent vendor or a contractor, skip the demo video. Ask three questions instead:
- Can I see the eval set, what is in it, and how were the examples chosen?
- What is the release gate, and does it run in CI on every change?
- When the agent fails, how do you reproduce the failure and prevent it permanently?
Teams that cannot answer these are demoing, not delivering. A demo shows the happy path with the lights on, the network healthy and the test account pre-seeded. An eval harness shows what happens when the locator is stale, the session expires, the API throttles and the agent is certain it finished the job.
A reasonable acceptance clause for any agent deliverable: no fixture, no fix. If a defect is found in acceptance testing, it becomes a fixture before it becomes a patch, so the fix is proven and the regression is gated forever. That single clause changes the incentive structure of the whole engagement, because every defect now makes the system permanently harder to break. It also protects you from the classic endgame where a vendor patches the symptom before the demo, the same failure resurfaces a month later, and nobody can prove the fix ever shipped.
Honest limits
State checks cover the happy path you recorded, and they cover it well. Novel failure modes, the ones nobody has imagined yet, still need human review. Fixtures also rot: when the application changes underneath them, the before and after snapshots no longer match reality, and a green gate becomes a false comfort. Gate maintenance is real work, and any team that tells you otherwise is selling something.
There is also a scope limit worth naming: state verification proves that the intended change happened. It does not prove the change was wise, compliant or reversible. An agent that deletes the right record for the wrong reason still passes the diff. Human review remains the control for intent, and intent is where the expensive mistakes live.
The correct posture is layered: state verification as the floor, human review for the unknown, and an honest admission that agents are probabilistic systems wearing the costume of determinism. None of this makes agents unusable. It makes them verifiable, which is the only version worth shipping.