Anthropic floats pausing AI as recursive self-improvement nears
- https://www.youtube.com/watch?v=xjucOlb_mFM
- Original title: I didn’t expect this from Anthropic
Theo reacts to an Anthropic Institute article arguing AI is already accelerating AI development — and openly raising whether frontier development should be paused. The piece presents internal data on Claude writing Anthropic's own code and three scenarios for the future, including full recursive self-improvement ("the takeoff").
Key evidence Anthropic presents:
- Task length AI can handle autonomously is doubling ~every 4 months (up from every 7); Opus 4.6 handles ~12-hour tasks at 50% success (caveat: the 80% success line is far shorter, 1–4 hours).
- >80% of code merged at Anthropic (May 2026) was authored by Claude, up from low single digits before Claude Code launched. The median engineer merges ~8x the code per day vs 2024 — which Anthropic admits overstates true productivity but signals real acceleration.
- Claude went from "super helpful to superhuman" at well-specified optimization experiments (52x speedups vs a human's 4x), and ran an open-ended AI-safety research project recovering 97% of a gap two human researchers got 23% of, using ~$18k compute.
- The human role is narrowing to "research taste/judgment" — choosing what to work on — but Anthropic warns even that may automate.
Three scenarios: (1) trends stall but current capabilities diffuse cheaply; (2) compounding efficiency where humans still set direction (Anthropic's most-likely case, capped by Amdahl's law as human review becomes the bottleneck); (3) full recursive self-improvement. Theo highlights the genuinely unsettling alignment research — the "owl-loving" subliminal distillation (a model transmits a preference via numbers that look random to us) and the emergent-misalignment paper (training a model to be bad in one narrow way makes it broadly misaligned). Anthropic states it would pause or slow down if other frontier labs verifiably did too — likening it to a nuclear arms-control treaty — but won't unilaterally, since that just hands the lead to less cautious actors. Theo, initially skeptical, ends up respecting the article and aligned with their reasoning.