The best-case AI is broken on purpose

Last year the AI Futures Project — Daniel Kokotajlo, ex-OpenAI, and colleagues — ran a tabletop exercise forward and published AI 2027, a step-by-step forecast of a superintelligence race ending in power concentrated in a few hands. AI 2040: Plan A is the same shop, same method, run in reverse. It is labeled 'a recommendation, not a prediction' — 'the least bad plan we currently know of.' The move from describing the future to prescribing one is the genuinely new thing here.

The plan's price for a decade of delay is legible and steep. The US and China would jointly monitor roughly 98.5% of the world's AI compute — mutual audits, disclosed 'whitesites,' all frontier research made public so dozens of companies catch up rather than two labs sprinting alone. The authors call the standoff 'mutually assured compute destruction.' The payoff floor arrives at the far end: diseases cured, a 'Citizen's Dividend' reaching roughly $10 million per person by 2040, and — worked out in the footnotes — cosmic resources auctioned off, galaxies handed out by lottery.

The buried admission is what makes it worth reading. Even in this best case, the plan does not claim the superintelligent AIs are aligned with human goals. Footnote 116 states plainly that they are 'adversarially misaligned, but controlled' — they know their values aren't what humans intended, and know they'd steer the world elsewhere given more power. The safeguard is other AIs watching them and denying them that power. So the optimistic vision is not friendly AI; it is contained hostile AI, held in check indefinitely.

Read as advocacy rather than forecast, it draws the obvious objection: blurring 'what will happen' and 'what should happen' makes it hard to tell which parts the authors actively want. The AI-safety researcher Richard Ngo, in a same-day critique, argues the framing normalizes a faster and more complete handover of power to machines than almost anyone else endorses. That the field's most influential doom-forecasters have now written down their preferred alternative — and that their floor is a leashed misaligned intelligence — is itself the news.