Here’s a line from last week’s AI news that made me smile: according to TechCrunch, Jellyfish reckons spending on agentic coding tools has grown 10x since the start of the year, and its chief scientist, Nicholas Arcolano, called it “growth that most CFOs didn’t plan for”. I bet they didn’t. And right on cue, Anthropic shipped the answer to the question every one of those CFOs is now asking: smart reports, in-app surveys and pooled group budgets for Claude Enterprise. In other words, your AI bill now comes with a report card.

I think that’s genuinely useful, and I also think most companies are going to read it the wrong way. Let me explain.

What Anthropic Actually Shipped

A smart report takes a team, a date range and the products you care about, samples the team’s transcripts and groups the work into “workstreams”. Each workstream shows what it cost, what it delivered and where it got stuck (Anthropic’s own examples are a missing connector or a failed tool call). The report ends with recommendations, such as turning a recurring workflow into a skill or adding a rule to a repository. The surveys pop up a short card after a reply, asking things like “what would slow down first if you lost access?”. And pooled budgets give a group one shared monthly budget, with individual limits kept as a guardrail and alerts at 50%, 75%, 95% and 100%.

Bear in mind the context, too. TechCrunch reports that Anthropic told investors its net revenue retention was 900% (existing customers spending nine times more than a year earlier), and that the same Jellyfish data puts these tools at about 2% of total engineering cost. So Anthropic has no trouble selling this stuff. What it does know is that the next budget conversation is coming, and it would rather be the one holding the spreadsheet when it does.

Tokens Are the New Lines of Code

Here’s the thing: the number everyone reaches for first is consumption, and consumption is the wrong number. Earlier this year Jellyfish looked at Claude Code usage and found the top 10% of developers burnt about 10 times more tokens than mid-level users, for roughly twice the output. Ten times the fuel, twice the distance. Arcolano’s advice was to track outcomes such as cost per pull request instead of total tokens, and I couldn’t agree more.

Us engineers have been here before. Remember when somebody decided lines of code was a good measure of productivity? Within a quarter, everyone wrote longer code. Put a token leaderboard on a wall (and give the “power users” a shout-out at the all-hands) and within a quarter you’ll get agents running in loops, five parallel attempts at the same ticket and nobody reading the four that got thrown away. Goodhart’s law doesn’t care whether the work is done by a human or a model.

So what should you measure? The things you already cared about before AI turned up: features shipped, lead time, incidents, cost per merged change, how long a customer waits for a fix. If the AI spend doubles and those don’t move, you’ve bought yourself a very expensive autocomplete. If they move and the spend stays flat, frankly, nobody in finance will ask for a token report.

The Friction List Is the Bit That Matters

The most valuable part of these reports, in my opinion, isn’t the cost column at all. It’s the friction: the missing connector, the failed tool call, the workflow where the agent gave up. Every one of those is wasted spend AND wasted time, and every one of them is an engineering problem, not a finance one.

I keep seeing the same pattern with the teams I talk to: the agent isn’t failing because the model is too dumb, it’s failing because the tools it’s given are vague, the docco is three versions out of date or the internal API returns an error message written for a human squinting at a log. Fix the tool description and the retries disappear (and so does a chunk of the bill). That’s why I’d argue agent experience now matters more than the polish of the buttons on your internal tools: your most frequent user is increasingly an agent, and it can’t squint. If a report tells you where your agents trip, treat it as a backlog, not a scorecard.

Pooled Budgets Are the Seat Model Waving Goodbye

Now the part most coverage skipped. A pooled group budget is a quiet admission that the per-seat licence doesn’t describe how this stuff gets used. One engineer runs ten agents overnight, another uses it twice a week for a regex (no judgement, regexes are hard). Charging both the same, or capping both the same, makes no sense, and the vendor knows it. Budgets move to the team, spend moves to usage, and I suspect the next step is pricing by task or outcome. If you sell software priced per head, this is the bit of the story I’d be losing sleep over.

There’s a catch, though. When the vendor writes the report, the vendor decides what “value” means, and 900% retention tells you how sticky that relationship already is. Use the reports (the insight is genuinely useful), but make sure your own numbers live in your own systems, in a form that would survive switching provider. Otherwise you’ll end up asking the people who send you the invoice whether the invoice was worth it.

So What Should You Do on Monday?

• Pick two or three outcome measures per team before you open any vendor dashboard, and agree them with finance so the conversation is about outcomes, not tokens.
• Read the friction list every month and turn the top items into work: better tool descriptions, missing connectors, rules in the repository, skills for the workflows people repeat.
• Move to team budgets rather than individual caps, and look at the outliers in both directions: the heavy users may have found something worth copying, or a loop worth killing.
• Don’t use any of this to rank individual engineers. Anthropic explicitly says the reports aren’t meant for that, and the moment people think they’re being watched, they’ll optimise for the report.

Bottom line: the AI bill finally comes with a report card, which is good news. Just remember who’s marking it, and mark your own homework too 😉