ARC-AGI-3 Measured the Harness, Not Just the Model

If you ship anything agentic, the settings in your wrapper are part of your benchmark score, and you probably never chose them deliberately. On 29 July 2026 OpenAI published that GPT-5.6 Sol went from 13.3% to 38.3% on the ARC-AGI-3 public task set without a single change to the weights (OpenAI). Two API settings did it. Here is what actually changed, why the comparison every outlet ran from it is invalid, and what to check in your own stack before you trust a score again.

ARC-AGI-3 is a benchmark in which AI agents must work out how unfamiliar 2D games function with no instructions given ([ARC Prize](https://arcprize.org/competitions/2026/arc-agi-3)). Scores use Relative Human Action Efficiency, a ratio against a human baseline rather than a raw solve rate.

We run the same agent workloads across models from more than one vendor, and every one of those runs passes through a wrapper we wrote ourselves, so in our own stack it is that wrapper, not the model card, that decides what survives from one step to the next. That is the part worth your attention. This is not a scoreboard dispute but a measurement problem that almost certainly exists in your own agent code.

What OpenAI actually changed

OpenAI changed two things and nothing else: it retained the model's private reasoning between actions instead of discarding it, and it replaced the harness's rolling truncation with compaction, which summarises old context rather than deleting it.

The official ARC-AGI-3 harness threw away private reasoning after every game action. The model could still see a record of its past moves, but not the plans or hypotheses that produced them, so each turn began by working out the game from scratch. On top of that, once history passed 175,000 characters the harness deleted the oldest messages outright (OpenAI). No durable reasoning plus a disappearing past is close to a worst case for a benchmark whose entire premise is learning an unfamiliar environment over time.

Both replacements are ordinary product settings available to any developer: reasoning items carried across turns (OpenAI docs) and compaction (OpenAI docs), both on the Responses API rather than legacy Chat Completions (migration guide). The reported effect was roughly triple the score with about six times fewer output tokens. Higher and cheaper at once is the detail that makes this credible rather than convenient: a setting that only inflated the score would be suspicious, while one that also cuts the token bill is doing real work.

The comparison everyone repeated is invalid

Here is where the coverage went wrong, and the error is subtle enough to be worth stating precisely.

The 38.3% figure is the public task set under OpenAI's own harness. The 30.2% figure attributed to Claude Opus 5 is an ARC-verified result under ARC Prize's standardised protocol. Different task sets and different harnesses, so the two numbers do not form a ranking.

Outlets ran the result as "38.3 percent, beating Opus 5's 30.2 percent" (The Decoder) and as settings lifting Sol "past Opus 5" (AI Weekly). Neither is supported. On the ARC Prize board, which is the only harness-controlled ranking that exists, Opus 5 sits at roughly 30.2% and Sol at roughly 7.8% (leaderboard, breakdown). Nothing published on 29 July 2026 changed a row on it.

Two further details keep the record straight. The like-for-like improvement is 13.3% to 38.3%, both on the public set, and that is the roughly threefold gain OpenAI claims. The 7.8% belongs to a different, semi-private measurement, and compounding the two produces a much larger and entirely fictional jump (Remio). Beyond that, the research post never mentions Opus 5 at all. The head-to-head framing came from an OpenAI employee on X and then from headlines, not from the write-up itself (breakdown).

One more figure belongs in view: 38.3% is still short of the roughly 48% human baseline OpenAI estimates from gameplay logs. The gap those settings closed is real, and the gap that remains is larger.

The case for the generic harness

The lazy conclusion is that ARC's harness was broken. It was not, and the counterargument deserves making properly.

ARC-AGI-3 uses an intentionally generic harness with no tools or vendor-specific features, on the reasoning that a simple runner makes model shortcomings visible and keeps comparisons fair. Commercial vendors tune harnesses per model; a benchmark that does the same becomes a contest between engineering teams rather than between models. The runner is public, so anyone can read exactly what it does (ARC-AGI-3-Agents) and play the same public games themselves (tasks, play surface).

François Chollet drew the line where I would draw it. Harnesses "custom-made to solve the benchmark or that contain knowledge about the benchmark format" are off limits; general-purpose API settings "that were not developed for ARC-AGI-3 and that are available to all API users" are fair game. He noted "a lot of back and forth with OpenAI about how to best test their models, especially with regard to compaction," and conceded that providers running different settings creates "a potential parity issue" — acceptable, in his view, "as long as the settings and the cost are clearly reported" (The Decoder). That is not a lab defending its turf. It is a benchmark operator conceding a methodological gap and describing the fix.

So both things hold. ARC's neutrality is a defensible design goal, and neutral defaults are not neutral in effect when one vendor's models are trained to carry reasoning forward and the default silently drops it. A generic harness does not measure models in a vacuum. It measures models under one particular memory policy, and that policy is itself a choice.

What to check in your own stack

The uncomfortable part is that almost nobody chose their memory policy on purpose. It arrived as a framework default.

Three things are worth checking. First, does your agent loop carry reasoning across tool calls, or does each step re-derive its plan from scratch? Second, when context fills, do you truncate or compact? Deleting the oldest turns is the cheapest thing to implement and the easiest to never revisit. Third, are you on the API surface your vendor actually deploys against, or on an older completions endpoint you adopted once and never migrated?

None of that is exotic. It is the same lesson we keep arriving at from other directions: throughput in agent systems is set by the surrounding machinery, not by picking a better model. It is why review capacity, not agent count, caps parallel work, why agents need a factory rather than a better prompt, and why a minimal harness sometimes beats a maximal one. It is also why we read the verification cost buried in a capability result rather than the headline number. A benchmark score is a property of a system, and the model is only one component of it.

The practical takeaway is not that ARC-AGI-3 is wrong or that one lab won. It is that a number you did not generate, produced by a harness you did not inspect, is not evidence about a model. Before a benchmark result changes a decision in your stack, find out what the runner did with the model's memory. If you want that layer audited properly, that is the work we do.

Frequently Asked Questions

Did GPT-5.6 Sol actually beat Claude Opus 5 on ARC-AGI-3? No. On the ARC Prize official board Opus 5 remains ahead at roughly 30.2% against roughly 7.8% (leaderboard). The 38.3% comes from a different task set under a different harness and does not rank against it.

What are retained reasoning and compaction? Retained reasoning carries a model's private reasoning items across turns instead of discarding them after each action. Compaction summarises older context when the window fills, rather than deleting the oldest messages (OpenAI docs).

Is using those settings cheating on a benchmark? Chollet's stated line is that benchmark-specific harnesses are off limits, while general-purpose API settings available to all users are fair, provided the settings and the cost are clearly reported (The Decoder).

Has the result been independently verified? Not yet. The 13.3% to 38.3% figure is self-reported by OpenAI on public test data (OpenAI). No independent semi-private re-run under documented setting parity has been published.

ARC-AGI-3 Measured the Harness, Not Just the Model