Grok 4.6 catches up on benchmarks but loses its edge
- YT :: https://www.youtube.com/watch?v=c7W8jpsjtCc
- Original title :: xAI just caught up (Grok 4.6 is here)
Grok 4.6 is a post-training refresh of 4.5 (xAI now owns Cursor's RL and post-training stack), aimed at long-running agents and multi-step work. It gains five points on the Artificial Analysis intelligence index to 61, putting it level with GPT-5.6 Sonnet, but it burns 30%+ more tokens per run, so it is both slower and roughly twice as expensive per task as 4.5. Theo's verdict: the two things that made Grok interesting — speed and cost — both regressed, so the model is less useful to him even though it is smarter.
What actually changed
No new pre-training. Longer supplemental training on curated model-generated data, regenerated SFT trajectories filtered by model-based checks, then agentic RL across coding, knowledge work, kernel optimization, webdev and CAD. The payoff is sustained multi-step work and more self-verification mid-run.
Benchmarks: intelligence index 61 (tied with 5.6 Sonnet, behind Fable 5 and Opus 5); DeepSWE 54% → 66%; Frontier Code 56.6 → 61.3; best-in-class on AA Briefcase, Harvey Lab and GDPval. Cursor Bench also jumped past 5.6 Sonnet, but that benchmark's data leaked into training earlier, so treat it as suspect.
Pricing is unchanged per token ($2/M in, $6/M out, 500k context) — cache reads went from 3¢ to 5¢/M. Cost per task is $0.84 vs $3.14 for Fable 5, so still cheap in absolute terms; the regression is relative to 4.5's unusual token efficiency, which was better than 5.6 Sonnet's and now is not.
Elon is already teasing 4.7 in 3-4 weeks, trained with a large dump of SpaceX company data.
Where it falls down
Design work is weak — the generated landing pages read as old-era AI slop (noise textures, brutalism, Tailwind-template card grids), and Fable, Opus and even 5.6 Sonnet are visibly better. Claude's design skill helps only marginally.
The Fish Slop game rebuild exposed worse problems: bad controls, wrong element sizing, wrong pacing in 2D; and the 3D port opened to a black screen — the first new model Theo has seen fail that task outright. After feedback plus a screenshot it "fixed" it into inverted left/right controls and misplaced ground geometry. Open-weight Kimi K3 and Muse 12 both do markedly better here.
Where it holds up
Real work in a real codebase went fine. A security audit of Theo's Lakebed project found the previous audit in git history and produced sensible findings (OAuth broker, worker isolation, identity tokens in URLs, sandbox-off flags in production) with no errors.
The ACP → Cursor SDK migration audit for T3 Code correctly identified the real trade-offs (SDK maintained by Cursor, no GUI approve/deny, no blocking questions, no reusable agent login). It then wrote a ~1000-line PR, babysat it, and stacked a second PR on top after mining Theo's local session history for unhandled events — competing contexts and multi-step orchestration held together.
The Grok Build CLI itself gets a genuine compliment; image rendering in the terminal is nice. Rough edge: it dropped into plan mode and lost the plan when the client disconnected.
Scorecard
Against Fable 5 and 5.6 Sonnet, Theo rates 4.6 a regression on cost (~6) and speed (~5.5), a modest bump on intelligence (~6.5), thoroughness and orchestration (~6.5). Grok's only leads were speed and price, and both moved the wrong way. Not a model he will keep using — but the rate of improvement makes 4.7 worth waiting for, and more cheap frontier competition is good for everyone.