Better models, worse tools

The assumption behind every model upgrade is that newer is strictly better. Armin Ronacher, relayed by Simon Willison on July 4, reports a case where it isn't. Wire Anthropic's latest models — Claude Opus 4.8 and Sonnet 5 — into a coding agent that isn't Anthropic's own, and give the model a code-editing tool shaped even slightly differently from the one it was trained on, and it starts making things up: inventing field names like requireUnique, oldText2 and newText2 that the tool doesn't accept, so the edit is rejected outright. In one agentic session that happened about a fifth of the time. None of the older Claude models did it.

stripping the model's own reasoning traces out of the history cut the failure rate roughly in half

The proposed explanation is that the newest models have been trained so hard on Claude Code's own native edit tool — flat file_path / old_string / new_string fields — that a tool holding the same intent in a different shape reads as off-distribution, and the model fights back toward the grammar it knows. The strangest supporting detail: the failure only appears after a real working session where the model has read files and reasoned about a problem, never on a fresh prompt. And stripping the model's own thinking traces out of the conversation history cut the error rate roughly in half — its accumulated reasoning is what pulls it back to its trained-in habits.

OpenAI is the clean counter-example to the same dynamic. Its Codex editing format, apply_patch — a structured diff with begin/end markers and context anchors instead of line numbers — used to live only inside its command-line tool. As of November it's a built-in API primitive you turn on with a single line, and OpenAI says it spent significant training compute making its models fluent in exactly that grammar. Same bet, opposite pole: lean all the way into one edit-tool shape rather than pretending the model is neutral about which one you hand it.

This is one developer's report, not a benchmark — the number is soft and context-dependent. But the implication for anyone building on a harness other than the model vendor's own is concrete: choosing a frontier model now means weighing raw capability against how closely your tool's schema matches the grammar that model was drilled on. The better-trained model can fight you harder precisely because its prior is stronger.