Of New Models Every Month and Knowing When Not to Upgrade

Posted by & filed under , .

A hand points at a glowing AI microchip surrounded by circuit lines on a dark teal background.Another week, another frontier model. Anthropic released Claude Fable 5.2 on 9 November, reportedly about 20% cheaper per token than Fable 5.1 and better on most benchmarks, and a few days later Google pushed out Gemini 4 Hydrogen and widened access to Gemini 4 Carbon. At this rate my AI tools ask me to upgrade more often than my phone does, and that’s saying something.

The question I keep hearing from founders and engineering leaders is “should we switch?” And my answer, more often than not, is a very unexciting “it depends what you’re using it for”. So let me explain what it depends on, with an example from my own code.

Not Every Job Needs the Smartest Model

In Calendrz we have a smart scheduler: an agent you ask, in plain English, to “find 30 minutes with Alice next Tuesday morning”, and it reads availability from your connected calendars and books the meeting. You’d think that’s the perfect place to throw the newest, cleverest model at the problem. But think about what the agent actually does: it reads a short prompt, calls a handful of well-described tools (find free slots, create an event), and writes a one-paragraph reply. That’s not a PhD maths problem. It needs to be reliable, fast and cheap, and it needs to follow the tool descriptions to the letter.

So in the code, the model is a configuration value, not something baked in. There’s an application-wide default, an admin can override it per user, and each user has a cap on how many model calls the agent can make a day, with a circuit breaker for when the provider has a bad day. Neither of the models we’ve configured is the newest frontier one, and for booking a 30-minute slot I’m not convinced it needs to be. When a new model comes out, trying it is a settings change, not a sprint.

How I’d Decide Whether to Upgrade

Here’s the checklist I’d use, as a CTO, every time a shiny new model lands:

  • What’s the job? Hard reasoning, long documents and big refactors benefit from the frontier. Routing, extraction and tool calling usually don’t, and a smaller model is faster too, which users notice far more than benchmark scores.
  • Do you have evals? If the only way to know whether the new model is better is “it felt better in a demo”, you don’t know. A few dozen real cases with expected outcomes will tell you more than any leaderboard.
  • What does it cost per task, not per token? A model that’s 20% cheaper per token but thinks twice as long can easily cost more per booked meeting. Measure the unit you care about.
  • How good are your tool descriptions? Honestly, in my experience, rewriting a vague tool description fixes more agent mistakes than a model upgrade. Agent experience is the new UX: the quality of your tool docs matters more than the polish of your buttons.

Don’t get me wrong, I’m not against upgrading. For hard coding work, the newest model often pays for itself in the first afternoon. The point is that “which model?” should be a decision you make per job, with data, and can reverse in five minutes. Not a religious conviction, and definitely not a three-week migration project.

Bottom line: the model release calendar isn’t going to slow down. Make the model a setting, keep some evals around, and let the new ones earn their place 🙂