The expert moat, blind-tested

Sixteen law professors from fourteen schools each wrote a short answer to forty real questions of the kind students bring to office hours — doctrine, hypotheticals, the policy reasoning that has no single right answer. Then they made nearly three thousand blind comparisons, picking the better answer without knowing its source. They chose the AI 75% of the time, and it beat all but one of them.

the preference for LLM answers is not reducible to length, clarity, or other stylistic markers — Salinas et al.

The assumed human moat in law was never recall of cases; it was judgment — the part everyone said machines couldn't do. This study put that belief in a controlled setting and the belief lost, on the experts' own questions, refereed by the experts themselves.

The most telling number is buried in the spread. The AI answers were flagged as pedagogically harmful 3.53% of the time; the professors' answers, 12.06%. But the human rate ranged from 1% for the best instructor to nearly 40% for the worst. The machine's advantage is not that it is brilliant — it is that it never has a bad day. The authors also checked whether the AI simply wrote prettier prose and found the opposite: the preference held after controlling for length and clarity.

Caveats are real and the authors press them: it is a preprint, one subject, one casebook, sixteen judges, and ninety-word written answers rather than a live tutor in a room. What it measures is the quality of a written answer, not whether you should hand a class to a chatbot. Even so, it relocates the burden of proof — the next person claiming AI can't handle expert judgment has to explain why a room full of contracts professors couldn't tell.