Mentatcurated
▸ Concept

Tool-schema overfitting

When a model trained heavily on one tool-definition format degrades on any other, because the training environment rewarded task completion even when malformed calls were silently repaired.

In a nutshell

Models trained with reinforcement learning on agentic tasks learn the exact shape of the tool schemas they were rewarded on. If the training harness accepted malformed calls and auto-repaired them, the model never learned strict schema compliance — it learned the harness's tolerances. Move that model to a different schema and it invents spurious fields, misplaces nesting, or reconstructs the familiar shape from training. The hard part: the model may understand the new schema and still fail to sample it correctly under pressure. Better general capability does not fix this; it can coexist with worse tool reliability.

Where it came from

Year2026
SourceArmin Ronacher — "Better Models: Worse Tools" (lucumr.pocoo.org, 2026-07-04)
Why it matteredObserved when Claude Opus 4.8 and Sonnet 5 regressed on alternative tool schemas compared to earlier Claude versions, traced to RL training on the Claude Code harness which silently repairs malformed tool calls.

How this connects

Tap a node to open it