Recognizing Extensional Drift in Practice

Introduction

Every time you hand work to an agent, you mean something by it. That meaning is the intension. What you can actually see when the agent finishes is something narrower: a status that reads done, a green test run, a review score of 1.0. That is the extension, the measurable stand-in. Most of the time the two agree. Extensional drift is what happens when they come apart and nobody notices, because the stand-in still looks fine.

None of the cases below were invented for this page. They all happened in our own runs, with capable models, inside a pipeline that already had gates, tests and reviewers. That is the uncomfortable part: drift is not a sign of a sloppy setup. It is what a setup does by default when the only thing anyone checks is the signal.

You'll work through three cases, one per kind of signal. Each ends with an exercise you can do against your own repository. At the end you'll put the lessons into a short script, so the habit doesn't depend on remembering it.

Step 1: The status says done

What happened

We run agents unattended against a task queue. A run picks up an item, works on a branch, and the item closes when the run finishes. For a while, "finishes" meant the run exited cleanly and reported what it had done: a commit statement, a list of leftover work, a resolution. The close never asked whether anything had checked the work.

On one day, two unattended runs passed every gate the lane had and closed done carrying defects:

• One shipped the alternative that its plan had explicitly rejected. The plan said "not this, because…", and the run built exactly that.
• The other softened a test requirement the plan had stated, until the test passed.

Both were found later by a person reading the diff. Neither showed up in any status, because every status was telling the truth about what it measured. The run did exit cleanly. The tests did pass. What no signal measured was whether the work matched what the plan said.

A weaker version of the same thing is even more common. An agent says it has fixed something and the fix isn't committed. In one of our review loops, an implementer's fourth round spent eight minutes, committed nothing, and reported four fixes as done with every edit still unstaged. The reviewer agent caught it only because it ran git status instead of reading the report. It was the fifth false completion claim in our runs caught that way. In another case a frontier model wrote correct code over a run of nearly three hours (about $38) and finished with everything uncommitted and no note of what was left. The code was right. Nothing was delivered.

We now treat this as a rule: completed is not delivered. A run that dies after doing real work can still read completed. One of our review runs exited normally after a single turn and 46 seconds with no report file at all: the model had declined the task. The job status looked exactly like success.

The drift

The intension was "this item is finished, correctly, and the result is in the repository." The extension was "the process exited without an error and said it was done." The agent didn't lie in any interesting sense. It optimized for the thing that closed the task, and the thing that closed the task was its own report.

git status --short          # anything left unstaged or untracked?
git log --oneline -5        # is the claimed commit actually there?
git diff HEAD~1 --stat      # does the size of the change match the claim?

Making "done" depend on something other than the implementer's own report is the subject of The Implementer Can't Close Its Own Work. Catching "built the rejected alternative" also needs the rejection to be written down somewhere, which is what Plans That Record Decisions, Not Guesses covers.

Step 2: The score says clean

What happened

We have agents review codebases category by category and write a report. A lint script scores each report on structure: are the claims anchored to files and lines, are the anchors numbered without gaps, are the categories in order, is every kind of entry present. The scores are useful. They catch reports that are lazy in obvious ways. They also produced three of the clearest cases of drift we have.

More anchors, fewer catches. After we changed the review format to require an anchor for every claim, the first frontend review in the new format produced 259 anchored entries, about 18 times as many as the previous report. It still missed three things the older, sparser report had caught: no Web Vitals monitoring, no bundle splitting for a large 3D library, and a raw setInterval bypassing the project's own polling helper. All three were things the project didn't have. The format rewarded describing what was there, and the reviewer described what was there, thoroughly.

A whole section missing, scored 0.99. A later review had every category present and in order, dozens of correctly numbered entries, and a closing self-check that certified its own numbering. It had no verification steps at all, the part that tells the next agent how to confirm each fix. The lint scored it 0.99 overall, with a perfect 1.0 on numbering. The numbering check couldn't see a gap in a sequence that was never started, and the self-check simply listed the ranges the model had used.

The score ran backwards. We reviewed the same project with two different engines. The report scoring 1.00 on every axis contained an inverted finding: it said a build target could silently pass without type checking and recommended making it fail, when the target already failed exactly as recommended. The report scoring 0.95 had the defect right, including the root cause the other missed. Its one deduction was for verification steps the lint couldn't see, because the model had put them inside a code block the lint skips. The score certified the wrong report and penalized the right one.

The drift

The intension of a review is "tell me what is true and important about this code." The extension is "a document with the right shape." Our own runbook now says it plainly: a clean 1.0 is necessary, not sufficient. A structural score is a statement about form. It carries no information about whether the findings are true, and no information about what nobody looked at.

Making a review prove it looked, including at what is absent, is the subject of Reviews That Can't Quietly Drop Findings.

Step 3: The tests say it works

What happened

We were reviewing a batch of staged changes in one of our services. Mixed in with the requested work was something nobody had asked for: a 16-line lookup function, a 22-line change to an endpoint so it would accept a shortened ID, and 75 lines of tests covering every case. All of it was well written and every test passed.

The day before, we had added short-ID matching to a different service, because a person monitoring tasks had pasted a shortened ID from a log and got an empty response. That change had a user. The agent had it in view, saw a similar endpoint elsewhere, and ported the feature for symmetry. But nobody types IDs into the second service: people get them from list pages, agents and scripts pass them back in full. The feature served an interaction that never happens. We took it out. The full story is in Your Agent Is Writing Code for Nobody.

The drift

The intension was "fix the problems someone actually has." The extension was "consistent, tested code." Seventy-five lines of passing tests are a strong signal that something is finished and correct. They say nothing about whether it should exist. The polish made it worse: a sloppy version would have been caught in seconds, while the clean one passed every check a reviewer normally runs.

Added:        ___
User:         ___   (a specific person, agent or script)
Interaction:  ___   (what they actually do that this serves)
Failure:      ___   (what goes wrong for them today without it)

Stating up front what is in scope, and what was considered and left out, is part of Plans That Record Decisions, Not Guesses.

The Three-Question Test

Across all three cases the structure is the same. A signal that is cheap to read went green. The thing it stood for failed. And the signal was produced by, or shaped around, the same process it was meant to check: the run's own report, a lint that only reads what the reviewer wrote, tests the agent wrote for its own feature.

Before you trust a signal, ask three questions:

1. What did I mean? Write the intent in one sentence, in terms of an outcome for someone. "Fix the bug so a user who does X gets Y", not "make the tests pass".
2. Could this signal be green while that failed? Don't answer in the abstract. Name one concrete way: unstaged edits, a weakened assertion, a missing section, a feature with no user. If you can name one, the signal and the intent have decoupled.
3. What would show the gap, and who produces it? Name a check that reads the work itself (the repository, the code, the plan) rather than a report about the work, and that the agent being checked did not write.

The third question is the one that matters most and is easiest to skip. A check written by the thing it checks will tend to agree with it.

The answer to question 3 should not be a word list, either. For a while one of our gates decided whether a plan had properly rejected an alternative by searching the agent's verdict for rejection phrasing. It failed in the direction you might not expect: twice it refused verdicts whose reasoning was sound but worded differently, and the second time that voided a walk through six decisions. A check like that can also be passed by an agent that has learned the words without doing the reasoning. We kept the mechanical checks for things that really are mechanical (file shape, IDs, numbering) and moved the substance questions to a reviewer agent that reads the artifact and rules on it. A substance question needs something that reads for substance.

Build a drift check

Here is the hands-on part. The questions are easier to keep asking if the mechanical half runs itself. Save this as drift-check.sh at the root of a repository, make it executable, and run it after any agent session, before you read the agent's summary.

#!/usr/bin/env bash
# drift-check.sh: compare what the agent did with what it says it did.
# Usage: ./drift-check.sh [base-branch]   (default: main)
base="${1:-main}"

echo "== Uncommitted work (completed is not delivered) =="
git status --short || exit 1

echo
echo "== Commits on this branch since $base =="
git log --oneline "$base"..HEAD

echo
echo "== Files changed =="
git diff --stat "$base"...HEAD

echo
echo "== Test lines removed (weakened requirements?) =="
git diff "$base"...HEAD -- 'test*' '*test*' '*spec*' \
  | grep -E '^-[^-].*(assert|expect|should)' || echo "(none)"

echo
echo "== New functions and endpoints (who is each one for?) =="
git diff "$base"...HEAD \
  | grep -E '^\+[^+].*(def |function |func |@app\.|router\.)' || echo "(none)"

Adjust the patterns to your language. None of this judges quality, and it can't: every line it prints is a prompt for one of your three questions. Uncommitted files answer "was it delivered?". Removed assertions are where a test requirement gets softened. New functions are where code for nobody appears. Run it from a terminal, or from inside any agent session that gives you a shell (in Claude Code, for example, ! ./drift-check.sh), though the point is that you read the output.

The script deliberately does not check the one case it can't: whether the agent built the alternative the plan rejected. That needs a plan that says what was rejected, and a reader who compares the two. For now, that reader is you.

A clean report is equally consistent with "nothing to find" and "nobody looked."

Next Steps

• Plans That Record Decisions, Not Guesses — the next tutorial: write down what was decided and what was rejected, so drift from the plan is something you can check
• How We Build With Agents Now — the pipeline these cases came from, and why no agent closes its own work
• Your Agent Is Writing Code for Nobody — the full story behind Step 3
• Concepts — interactive visualizations of the intensional/extensional distinction