To compare AI models fairly, use 10–20 of your own real tasks, run each model three times with identical settings, hide which model wrote what, score every output as accept, edit, or reject using rules you wrote first, then divide each model’s total cost by its accepted outputs. Benchmarks, single runs and gut feeling all mislead: identical settings still give different results, and experienced developers misjudged their own speed in a controlled trial. To include locally-run models in your comparison, see our guide to running local AI models.
Two models list at the same $2 per million input tokens and $10 per million output tokens, yet one composite index puts them four points apart, while the other lab’s launch page says its model wins on the benchmarks it chose. We went through that exact case in our GPT-6.1 Sol review. Neither number tells you which model drafts your client emails better or catches errors in your spreadsheet formulas. Only your own work can. An honest test takes one afternoon and a spreadsheet.
Why you can’t compare AI models with a leaderboard alone
A composite index averages many tests, and a launch page shows the ones the vendor wins. Both can be accurate and still disagree. That is the pattern in our model rankings piece, where the same model’s score on the same benchmark differed by about 20 percent between trackers within two weeks. Launch-week numbers are also self-reported, as we saw when three flagship models arrived within nine days.
List price is not your bill either. Artificial Analysis lists GPT-6.1 Sol at $0.72 per task on its index, with 67 million output tokens used across the whole run against a median of 81 million for comparable models. Two models with the same per-token price can still produce different bills, because they use different numbers of tokens to do the same job.
Step 1: Collect 10 to 20 real tasks
Pull them from the last two weeks of your actual work: emails you wrote, documents you summarized, formulas you fixed, product descriptions you drafted. Include two or three easy ones, a handful of typical ones, and two or three that were painful. Save each input exactly as you gave it.
- Include two or three tasks where you already know the right answer.
- Remove client names and confidential data first, and check the tool’s training default before you paste anything. Our usage-policy template covers the rule in five lines.
- Keep the arithmetic in mind: with 10 tasks, one task is 10 points of your score; with 20, it is 5. A one-task gap is a tie, not a winner. A software team would use far more cases. For a one-person decision, 10 to 20 is a floor.
Step 2: Freeze everything except the model
Use the same wording for every model. Write down the exact model ID, the date, the reasoning or effort setting, and whether tools such as web search were on. Effort changes both cost and speed: Artificial Analysis lists GPT-6.1 Sol at maximum effort at about 51 tokens per second, which is why hidden reasoning tokens belong in any cost comparison.
Prompt wording matters more than it should. A 2026 review of agent-evaluation methods summarizes earlier studies in which meaning-preserving changes to prompt formatting moved accuracy by a median of 7.5 points on mostly classification tasks, and individual templates reversed which model looked better. Those are not results for your emails, so treat them as a prompt to check: if two models finish close, reword the prompt once and see whether the same model still leads.
Step 3: Run every task three times
One run is one draw. Across 60,000 coding-benchmark runs, the review reports that a single run’s pass rate varied by 2.2 to 6.0 percentage points, and that even at temperature 0 the standard deviation stayed above 1.5 points because of infrastructure nondeterminism. It adds that gains of two to three points, a common size in published comparisons, fit inside that noise. In another study it cites, two identical systems looked significantly different in up to 26 percent of single-score comparisons.
Those figures come from coding benchmarks, not from your work. The lesson that transfers is that you need your own spread, so run each task three times per model, each in a fresh conversation so earlier answers can’t shape later ones. Fifteen tasks on two models is 90 outputs. Plan on roughly a minute each to score, and treat that as a planning guess.
Step 4: Hide the model names and shuffle the order
Your sense of which output is better, and how much time it saved, is unreliable. In a randomized trial by METR, 16 experienced open-source developers forecast that AI would make them 24 percent faster; afterwards they believed it had made them about 20 percent faster; the clock showed tasks taking 19 percent longer, with a confidence interval from 2 to 39 percent (METR’s paper). We covered that result in our productivity tools guide.
Don’t read it as proof that AI slows you down. METR’s own 2026 follow-up (57 developers, 800-plus tasks) estimated an 18 percent slowdown for returning developers (interval from 38 percent slower to 9 percent faster) and 4 percent for new ones, then said selection effects made the data “very weak evidence” and that today’s speedup is probably higher. It also found that time tracking breaks down when people work on something else while an agent runs. The part that held up is that self-reports are unreliable, yours included.
- Give each task two output cells, “Output 1” and “Output 2.”
- In a key column, use =IF(RAND()>0.5,”X”,”Y”) to decide which model goes first, then copy the column and paste it as values so it stops changing.
- Hide the key column until every output is scored.
Order matters even for machines: one AI judge in a 2026 study picked the first-shown answer as the majority winner on 72 percent of questions. Shuffle anyway.
Step 5: Score with three verdicts you defined first
Before you look at any output, write one line per task saying what “accept” means. Then give every output one of three verdicts:
- Accept: usable as is.
- Edit: usable after a quick fix. Log the minutes it took.
- Reject: not worth fixing.
Check facts, figures and quotes before you accept anything, using the failure-mode checklist in our fact-checking guide.
Resist handing the scoring to another AI. A study that repeated the same judging task 50 times found that the same AI judge flipped its verdict on an identical pair 13.6 percent of the time on average, and more than 20 percent of the time on 28 percent of questions. It also picked confident winners where its own scores showed almost no difference, and reworded judge prompts changed the majority verdict in a quarter of the tested cases. The study used two OpenAI judges, 29 questions and deliberately close pairs, so treat it as a warning, not a measurement of every judge. If you use an AI judge at all, use it to pre-sort, run it several times (the study needed 11 trials, on average, to match a 50-trial verdict with 95 percent probability), and check a sample by hand.
Step 6: Price the usable outputs, not the tokens
Your own editing time usually outweighs the token bill, so count it. The formula:
Cost per usable output = (total spend on all runs + edit minutes × your hourly rate ÷ 60) ÷ (accepted + edited outputs)
Here is the arithmetic with invented numbers, not results. Say you value your time at $30 an hour, or $0.50 a minute, and each edit takes four minutes.
| Invented example | Model X | Model Y |
|---|---|---|
| Runs (15 tasks × 3) | 45 | 45 |
| Accepted / Edited / Rejected | 33 / 8 / 4 | 29 / 10 / 6 |
| Token spend, all runs | $4.50 | $2.70 |
| Edit time at $0.50 a minute | 32 min = $16.00 | 40 min = $20.00 |
| Total cost | $20.50 | $22.70 |
| Usable outputs | 41 | 39 |
| Cost per usable output | $0.50 | $0.58 |
Model Y spent $1.80 less on tokens and still cost more per usable output, because you spent your time fixing it. Time yourself on reviewing and editing only, not on waiting for the answer.
If you test through a subscription rather than an API, there is no per-token bill. Compare the plans first with our API vs. Plus vs. Business guide, then judge the usable-output rate alone.
Consider adding a cheap baseline column: the least expensive tier, or the cheaper model asked twice. In one re-evaluation of published agent designs, simply retrying the model matched more elaborate setups at a fraction of the cost, though that result came from a function-level coding benchmark, so test it on your own tasks.
Step 7: Decide with a rule you wrote before the results
Write the switching rule first, so the numbers can’t redefine what counts as a win. An example: “Switch if the new model gets at least two more of ten tasks accepted, averaged over the three runs, or costs 25 percent less per usable output at equal quality. Otherwise stay.” The thresholds are examples; adjust them to your stakes. If the models tie, pick the cheaper or the more reliably available one, and keep your task file. Models change every few weeks, and rerunning the same ten tasks next quarter takes far less time than building them.
Spreadsheet columns to copy
| Column | What goes in it |
|---|---|
| Task ID | A short label, plus the saved input |
| Model (hidden key) | X or Y; hidden until scoring ends |
| Run | 1, 2 or 3 |
| Settings | Model ID, effort setting, date, tools on or off |
| Output | The text, pasted unedited |
| Verdict | Accept, Edit or Reject, against your pre-written rule |
| Edit minutes | Review and fix time only |
| Cost | From the usage page, per run or per batch |
When this method is the wrong tool
This is a decision aid for small, low-stakes choices, not statistical proof. It tells you about your tasks, with these settings, this month. Don’t use it alone for legal, medical or financial work, where an expert should review the outputs. And if quality only shows up later, as with ad performance or customer replies, you need a live A/B test over weeks, not a spreadsheet.
Frequently Asked Questions
How many tasks do I need to compare AI models?
Ten to twenty real tasks is a workable floor for a one-person decision. With 10 tasks, one task changes your score by 10 points, so treat a one-task difference as a tie. Software teams building products typically use many more.
Do I really need to run each task three times?
Yes, if the models finish close. Studies of coding benchmarks found that identical settings still produce different results, with single-run pass rates varying by several percentage points. Three runs per task shows you your own spread before you credit a small difference.
Can I use an AI to score the outputs?
Only as a first sort. In a repeated-trial study, the same AI judge changed its verdict on the same pair 13.6 percent of the time on average, and rewording the judge prompt changed the majority outcome in about a quarter of tested cases. Check a sample by hand.
Are benchmark scores useless, then?
No. Use them to build a shortlist of two or three models, then test those on your own work. Benchmarks tell you who is worth testing, while your tasks tell you who to pay for.
How often should I repeat the test?
When a model you use is replaced, repriced or upgraded, and at least once a quarter. Keep your task file and scoring rules so a rerun takes a fraction of the first afternoon.
What if two models tie?
Choose the one with the lower cost per usable output, or the one whose pricing and availability look more stable. A tie on your tasks means the choice is yours, not that you made a mistake in the test.
Shurah is the founder of AI Tools Daily, tracking pricing, licensing and policy changes across AI tools so readers can make decisions without wading through marketing claims themselves.
