I Deleted My Own AI Comparison Table. Here's What Replaced It.

Confession first: the previous version of this post had a neat table: Claude Opus 5 vs GPT-5.6 Sol, percentages to one decimal place, presented like I'd measured them. I hadn't, not verifiably. I'd absorbed lab reports, marketing, and vibes, and laundered them into fake precision. That's exactly the slop I'm keeping off this blog, so I deleted it. This is the honest version, plus everything I’ve thought since about why model comparisons are broken in principle, not just in my execution.

Why nobody's table can answer your question

Every comparison was run on someone else's tasks. But you don't employ a model for puzzles. You need it to survive your repository: 45,000 lines of old decisions, threading bugs, strict formats. A puzzle champion can still be the model that confidently rewrites your API and breaks everything downstream.

The differences that matter are behaviors, not scores. When a command fails, does it read the error and adapt, or retry the identical broken thing? Does a refactor preserve contracts or silently "improve" them into breakage? Does it invent files that don't exist? No leaderboard measures any of this on your code. When I need release facts rather than vibes, I check ModelRegistry, an open index of frontier models I maintain. But facts are not verdicts.

The dimension benchmarks never measure: taste

Here's the thought I've never seen in a comparison post. Two models can both produce correct code where one is clearly written by the better engineer: names that explain themselves, functions that end where they should, no cleverness where clarity would do, tests that pin the actual risk. Review a hundred AI diffs and you develop a nose for this, and the models differ enormously at it. Yet no benchmark scores elegance, because elegance resists automation. It's judged, not measured.

This matters more than it sounds, because AI-generated code becomes your codebase. An ugly-but-correct diff costs you every future reading; a tasteful one pays rent forever. When you run your own comparison, add a fourth scorecard beyond pass/fail/cost: would I be proud to own this diff in two years? The model that writes code you'd sign your name to is worth real money over the one that merely compiles. A premium no leaderboard prices in.

The weekend experiment (and when to skip it)

Snapshot your repo. Give both models the same three jobs: a real multi-file refactor (then actually compile it), a task designed to fail first (then count retries to green), a strict-format task on messy input. Score compiling, contract preservation, retries, cost, plus the taste review above.

And the meta-honesty: for most everyday tasks, both models will do fine, and the comparison doesn't matter. The obsession with picking a winner is itself the trap. Teams burn weeks benchmarking while their guardrails, evals, and review discipline (which transfer across every model, forever) rot untended. My default: Opus-class as careful architect for deep refactors, Sol-class as fast explorer for sweeps, cheap small models on syntax. A starting hypothesis from my repos, worth nothing on yours until tested. Anyone selling a universal winner is selling something, including, on reflection, the version of me that sold you a table.