the wrong scoreboard.
Most AI conversations get stuck on capability. Look what it can write. Look what it can summarize. Look, it passed the bar exam. It's genuinely impressive, and it's the wrong thing to measure.
Capability is what the tool can do in a demo. That has almost nothing to do with whether it's worth having. A Formula 1 car is extremely capable. It is also a terrible way to get your kids to school.
The question that actually matters is quieter. After you rolled this thing out, what did a real person stop doing?
the only metric I trust.
If nobody's day changed, you didn't deploy AI. You bought a subscription and a story to tell investors.
A tool that adds a capability but subtracts nothing from anyone's week is not a win. It's a new tab, one more thing to check, and a bill.
So when I evaluate whether something's working, I go find a person and ask what fell off their plate. Not what's now possible. What's now gone.
Did the analyst stop rebuilding the same deck every Friday. Did support stop writing the same answer to the same question for the ninetieth time. Did the ops lead stop spending her Monday morning reconciling two lists that should have matched on their own. Those are real answers. "It can draft emails" is not.
why this reframe changes what you build.
When you grade by what people stop doing, your whole approach shifts.
You stop chasing the flashiest use case and start hunting the most repetitive one, because repetition is where time actually hides. You stop asking "what's the coolest thing this can do" and start asking "what's the dumbest thing my best people are stuck doing." The second question is less fun at a conference and worth far more on a Tuesday.
It also keeps you honest about the ugly cases. A tool can look like a triumph and quietly create work, if it produces output someone now has to check, clean, and fix. Something that hands you a confident draft you can't trust hasn't saved you the task. It's just moved you from writing to auditing, and auditing is often the slower job.
So here's the whole thing on a sticky note. Don't ask what it can do. Walk over to the people it was supposed to help and ask them what they stopped doing this month. If the honest answer is nothing, you have your review, and it doesn't matter how well it did on the bar exam.