Custom Eval Harness vs Published Benchmark Scores: Which Should Decide Your Model?
GPT-5.6 Sol scored 7.8% and 38.3% on the same ARC-AGI-3 tasks. Only the harness changed. When to trust leaderboards, when to build your own eval.
Use published benchmarks to shortlist, use your own harness to decide, and never let the two swap roles. The July 2026 ARC-AGI-3 dispute is the cleanest evidence anyone has produced for this. OpenAI's position — "benchmarks never measure just the model but also the technical setup around it" — is correct. ARC Prize's position — that official scores must use a standardised approach without provider-specific settings so that cross-vendor comparisons stay honest — is also correct. François Chollet drew the line that actually resolves it: harnesses built to solve the benchmark, or that encode knowledge of its format, are off limits; general-purpose API features available to every customer are fair game, provided the settings and the cost are reported. That is a rule you can apply to your own evaluations tomorrow. For a buying decision, the practical consequence is a sequence, not a choice. Leaderboards eliminate candidates quickly and cost nothing: if a model is not in the conversation on public reasoning or coding benchmarks, no amount of scaffolding will rescue it. But once two or three candidates sit within a few points of each other, the published number stops discriminating and your harness starts. A roughly fivefold score swing from two API flags is larger than almost any gap between frontier models on a leaderboard, which means the scaffolding decision dominates the model decision at exactly the moment teams stop thinking about it. Cursor's engineering data points the same way from a completely different direction: cloud agents went from roughly one in ten merged pull requests in December to more than half by July, and the company credits the environment — giving agents their own machines and letting them repair their own setup — rather than a better model. Two independent 2026 data points, one conclusion: the system around the model is the variable that moved. There is one case where published scores should win outright, and it is not a technical one. If a number has to survive a board paper, a procurement process or a regulator, a self-reported internal result will not carry it. ARC Prize publishes its funding, its conflict-of-interest recusal rules and an independent academic panel precisely so its numbers hold up under that scrutiny. Your harness has none of that, and it should not pretend to. Our recommendation: shortlist on public benchmarks in an afternoon, then spend two weeks building a harness of 30 to 50 real tasks pulled from your own logs, with per-task token and cost logging and at least one scaffolding flag you can toggle. Rerun it on every model change and every loop change. If you only have budget for one of the two, build the harness — but publish the settings and the cost alongside the score, the same standard you would demand of a vendor.
Detailed Comparison
A side-by-side analysis of key factors to help you make the right choice.
| Factor | Custom Eval HarnessRecommended | Published Benchmark Scores | Winner |
|---|---|---|---|
| What the number actually measures | The model plus your scaffolding as a single system: your prompts, your state handling, your tools, your retry policy. | The model under a provider-neutral setup that deliberately excludes vendor-specific API features. | |
| Comparability across vendors | Only as fair as your own wrapper. OpenAI's harness used per-model settings, which is exactly what makes its 38.3 percent incomparable to a standardised run. | The entire point. ARC Prize runs every model through a standardised approach without provider-specific settings. | |
| Predicts what you will actually ship | High. Sol moved from 7.8 to 38.3 percent on the same public tasks with no change to the weights, purely by retaining reasoning and compacting instead of truncating. | Low for agentic work. In the official harness the model's reasoning is discarded after every action step, which is not how anyone runs an agent in production. | |
| Effort to get a usable number | Days to weeks: a task set, graders, budget instrumentation, and someone who keeps it honest as the codebase moves. | Minutes. Leaderboards are published, dated and free to read. | |
| Reveals how much your loop matters | Directly. Two Responses API settings moved one model's score roughly fivefold and cut output tokens sixfold at the same time. | Invisible by design. A standardised harness holds scaffolding constant, so the largest single lever in your stack never appears in the number. | |
| Auditability and conflict-of-interest control | Whatever you enforce yourself. Internal harnesses are rarely reviewed and easy to tune, consciously or not, toward the model the team already prefers. | Published policy. ARC Prize documents its funding, requires recusal from staff holding equity in a tested lab, and submits its methodology to an independent academic panel. | |
| Reproducibility by a third party | Internal only. Nobody outside the team can rerun it, and a vendor claim measured this way stays self-reported — the 38.3 percent has had no independent private-set verification. | Reproducible by the publisher on a dataset and a publication cadence committed to in advance. | |
| Cost visibility | Your real invoice on your real tasks, including retries and the tokens your loop wastes. Sol at 5 / 30 dollars per million and Opus 5 at 5 / 25 differ far less than the token counts two harnesses produce. | Published but generic. The ARC-AGI-3 leaderboard plots cost per task and only lists systems costing under 10,000 dollars to run — useful signal, not your bill. | |
| Relevance to your workload | You choose the tasks, so the score is about your domain, your documents and your failure modes. | Abstract by construction. ARC-AGI-3 tests on-the-fly adaptation to novel interactive environments; the average human tester scores 48 percent. | |
| Total Score | 4/ 9 | 4/ 9 | 1 ties |
Key Statistics
Real data from verified industry sources to support your decision.
MLQ.ai
THE DECODER
MLQ.ai
MLQ.ai
Cursor Engineering Blog
ARC Prize
OpenRouter Models API
All statistics come from verified third-party sources. Source, year, and direct link are shown on each metric.
When to Choose Each Option
Clear guidance based on your specific situation and needs.
Choose Custom Eval Harness when...
- You are shipping an agent rather than a chat box: state handling, compaction and retries are part of your product, and a score measured without them says nothing about your latency or your bill.
- Two candidate models sit within a few points of each other on public leaderboards — at that distance the harness is a bigger lever than the model.
- Your domain is narrow (claims handling, contract review, internal tooling) and no public benchmark covers the work you actually do.
- You need a defensible cost-per-task figure for a budget or a client, measured on your own tasks rather than on someone else's.
Choose Published Benchmark Scores when...
- You are at the shortlist stage and need to eliminate clearly unsuitable models before spending engineering time on any of them.
- You need a citable third-party number for a board paper, a procurement document or a regulatory filing — self-reported internal scores do not survive that scrutiny.
- You are tracking generational progress over time rather than choosing which model to run this quarter.
- Nobody on the team owns evaluation yet: an unmaintained internal harness produces worse decisions than a published leaderboard.
Our Recommendation
Use published benchmarks to shortlist, use your own harness to decide, and never let the two swap roles. The July 2026 ARC-AGI-3 dispute is the cleanest evidence anyone has produced for this. OpenAI's position — "benchmarks never measure just the model but also the technical setup around it" — is correct. ARC Prize's position — that official scores must use a standardised approach without provider-specific settings so that cross-vendor comparisons stay honest — is also correct. François Chollet drew the line that actually resolves it: harnesses built to solve the benchmark, or that encode knowledge of its format, are off limits; general-purpose API features available to every customer are fair game, provided the settings and the cost are reported. That is a rule you can apply to your own evaluations tomorrow. For a buying decision, the practical consequence is a sequence, not a choice. Leaderboards eliminate candidates quickly and cost nothing: if a model is not in the conversation on public reasoning or coding benchmarks, no amount of scaffolding will rescue it. But once two or three candidates sit within a few points of each other, the published number stops discriminating and your harness starts. A roughly fivefold score swing from two API flags is larger than almost any gap between frontier models on a leaderboard, which means the scaffolding decision dominates the model decision at exactly the moment teams stop thinking about it. Cursor's engineering data points the same way from a completely different direction: cloud agents went from roughly one in ten merged pull requests in December to more than half by July, and the company credits the environment — giving agents their own machines and letting them repair their own setup — rather than a better model. Two independent 2026 data points, one conclusion: the system around the model is the variable that moved. There is one case where published scores should win outright, and it is not a technical one. If a number has to survive a board paper, a procurement process or a regulator, a self-reported internal result will not carry it. ARC Prize publishes its funding, its conflict-of-interest recusal rules and an independent academic panel precisely so its numbers hold up under that scrutiny. Your harness has none of that, and it should not pretend to. Our recommendation: shortlist on public benchmarks in an afternoon, then spend two weeks building a harness of 30 to 50 real tasks pulled from your own logs, with per-task token and cost logging and at least one scaffolding flag you can toggle. Rerun it on every model change and every loop change. If you only have budget for one of the two, build the harness — but publish the settings and the cost alongside the score, the same standard you would demand of a vendor.
Frequently Asked Questions
Common questions about this comparison answered.
Need help deciding?
Book a free 30-minute consultation and we'll help you determine the best approach for your specific project.