Recognizing Extensional Drift in Practice

Introduction

Extensional drift is what happens when you optimize for something measurable and quietly lose the thing you actually cared about. The metrics improve. The dashboards look healthy. But the original purpose — the reason you built the thing — has slipped away.

This isn't a failure of tools or intelligence. It's a structural tendency. Anything that can be measured can be optimized, and anything that gets optimized tends to become the goal — even when it was only ever meant to be a proxy for the goal. If you've worked with AI systems, you've seen this. If you've managed people, you've seen it there too. The pattern is the same.

In this tutorial, you'll work through three concrete scenarios where extensional drift shows up in everyday technical work. Each one includes an example to examine, a question to answer, and a reflection prompt. The goal isn't to memorize rules — it's to build the habit of noticing.

Step 1: Drift in Code Review

Scenario

An AI coding agent has been asked to improve error handling in an API endpoint. Here's the original code:

async function getUserProfile(userId) {
  const user = await db.users.findById(userId);
  if (!user) {
    throw new NotFoundError(`User ${userId} not found`);
  }
  return user;
}

The agent produces this "improvement":

async function getUserProfile(userId) {
  try {
    const user = await db.users.findById(userId);
    if (!user) {
      return { success: false, data: null, error: "User not found" };
    }
    return { success: true, data: user, error: null };
  } catch (err) {
    console.log("Error fetching user:", err.message);
    return { success: false, data: null, error: "Internal error" };
  }
}

At first glance, this looks more robust. There's a consistent return shape. Errors are caught. Nothing throws. A linter would be happy. Tests checking the response structure would pass.

But look at what changed. The original code used typed errors (NotFoundError) that callers could distinguish and handle differently. The new code flattens every failure into a generic string. A 404 and a database connection timeout now look identical to the caller. The catch block silently logs and returns, so a corrupted database connection won't trigger any alerts. The userId that would help with debugging has been stripped from the error message.

The agent optimized for the extension — a function that never throws, always returns a consistent shape — and lost the intension: error handling that preserves diagnostic information and lets callers make informed decisions.

Step 2: Drift in Prompt Engineering

Scenario

A team is building a customer support chatbot. Their initial prompt includes this instruction:

When a customer describes a problem, acknowledge their
frustration before suggesting solutions. Be empathetic
and human in your responses.

After a round of A/B testing, the team notices that shorter responses get higher satisfaction scores. So they revise:

Keep responses under 3 sentences. Lead with the solution.
Use positive language. End with "Is there anything else
I can help you with?"

Satisfaction scores go up. Resolution time goes down. The metrics say this is better.

But something has shifted. The original prompt encoded a relationship between the bot and the customer: listen first, then help. The revised prompt encodes a transaction: deliver the answer, move on. Customers who had straightforward questions are happier. Customers who were upset and needed to feel heard are now getting efficient, emotionally flat responses — and they're not showing up in the satisfaction scores because they've stopped engaging entirely.

The metric (satisfaction score) drifted from its intension (customer feels supported) because the measurement only captured people who stayed in the conversation. The prompt was optimized for the metric, not for the purpose the metric was supposed to represent.

Step 3: Drift in Metric Design

Scenario

A machine learning team is building a content moderation system. They start with a clear goal: reduce harmful content on the platform. They choose precision and recall on a labeled dataset as their primary metrics.

The model performs well: 94% precision, 91% recall. The team ships it. Over the next quarter, they optimize further — tuning thresholds, adding edge cases to the training set, adjusting class weights. Precision hits 97%. Recall hits 95%.

But user reports of harmful content haven't decreased. Why?

The labeled dataset reflected last quarter's patterns. Bad actors adapted. They're now using techniques the training data doesn't cover — subtle context manipulation, coded language, content that's harmful in combination but benign in isolation. The model's precision and recall on the test set are excellent. Its performance on the actual goal — reducing harm — has stalled.

This is extensional drift at the measurement layer. The team optimized the numbers that were easy to compute (model performance on a static dataset) and lost track of the thing those numbers were meant to represent (real-world safety). The metrics became the goal.

When a measure becomes a target, it ceases to be a good measure. — Goodhart's Law, generalized

The Three-Question Test

Across all three scenarios, the same structure appears. Something measurable improves. Something meaningful degrades. And the degradation is invisible to the measurement system.

The three-question test from Step 3 generalizes to any situation where you suspect drift:

1. What was the original intent? Not the metric, not the spec — the reason behind them.
2. Can the metric improve while the intent degrades? If yes, the metric and the intent have decoupled.
3. What signal would reveal the gap? This is your early warning system.

This isn't a formula you apply once. It's a habit of asking "what did we actually mean?" every time something looks like it's working. The more automatic the optimization — and AI makes optimization very automatic — the more important this question becomes.

Next Steps

For more on the intensional/extensional distinction and how it applies to AI systems, explore:

• Concepts — interactive visualizations of probability paths and blind spots in multi-agent systems
• Human-AI Collaboration: When Your Agent Gets Pedantic — a deeper look at how AI agents default to extensional thinking
• Blog — ongoing writing about intensional reasoning, alignment, and collaboration