Measuring AI by outcomes, not demos

The most common reason an AI initiative loses funding is not that it failed. It is that nobody could say clearly what it changed. Model accuracy improved, the demo impressed the board, and twelve months later no operational number had moved in a way anyone could attribute to the work.

Start from a metric that already exists

Before building anything, find the number the business already reports and already cares about. Average handling time. Days sales outstanding. Order accuracy. Time to first response. If the project cannot be tied to a metric that predates it, you will be arguing about value from a position of weakness for the rest of its life.

Then capture the baseline properly, including its variance. A single month's figure is not a baseline; it is a data point. Knowing the normal range is what lets you tell improvement apart from noise later.

Separate model metrics from business metrics

• Model metrics (precision, recall, groundedness) tell engineers whether the system works
• Business metrics (cost, cycle time, error rate, revenue) tell leadership whether it matters
• Adoption metrics (usage, override rate, abandonment) explain the gap between the two

That third category is the one teams skip, and it is usually where the answer lives. A model with excellent offline accuracy that users override half the time is telling you something precise: it is accurate on the wrong distribution, or the interface makes accepting its output harder than doing the work manually.

Prefer a holdout to a before-and-after

Before-and-after comparisons absorb every other change in the business. Run the new system against a holdout group and the attribution argument ends before it starts.

This is not always possible, but it is possible far more often than teams assume — by region, by team, by document type, by customer segment. The cost of setting one up is small. The cost of not being able to defend a result is an entire programme.

Report in the language of the operation, not the language of the model. Not 'F1 improved to 0.91' but 'exceptions requiring manual review fell from 340 to 96 per week'. The second sentence survives a budget meeting. The first does not.