Verified World Models for Interactive Reasoning in ARC-AGI-3

ft09: 6 of 6 levels, 75 actions

sb26: 8 of 8 levels, 124 actions

01Problem statement

In ARC-AGI-2 a solver receives several solved examples and one test input. ARC-AGI-3 removes the examples. The agent receives a 64×64 frame with 16 colours and a small action set: directional actions, a few buttons, a click at a chosen coordinate, and a reset. It must determine, by acting, which parts of the frame belong to the game world, which parts are counters or legends, what each action does, and what condition completes a level. Levels within one game share their mechanics and increase in difficulty.

The official metric is relative human action efficiency (RHAE) [7]. It compares the number of real actions used on each level with the actions of a first-time human player, penalises the excess quadratically, gives no credit for a level that is not finished, and weights later levels more heavily than earlier ones. A wasted action is therefore expensive, and knowledge that carries over to later levels is valuable. Twenty-five games have been released publicly. The competition itself runs as a Kaggle code competition on hidden games with internet access disabled, which means that a submitted agent must carry its model with it.

ft09, levels 1 to 6

sb26, levels 1 to 8

02Prior work and what this project borrows

This project did not invent its central idea. It belongs to the family of executable world models: a language model writes a program that describes the environment, the program is tested against what has actually been observed, and planning happens inside the tested program. WorldCoder [3] introduced this for structured environments. Two systems for ARC-AGI-3, Schema [1] and Retrodict [2], are the direct sources of the design described below, and Tycho [4] belongs to the same family. An ablation study [5] reports that an executable model that is not verified by replay can perform worse than plain text notes, which is why verification against the record is the centre of this design.

Schema

Schema (Zeng et al., Impossible Research, UC Berkeley and CMU) treats the agent as a physicist. The agent must decide what the pixels are and how they move, and it keeps both decisions in one editable Python program with three functions: parse_obs (what the world is), step (how it changes) and is_goal (what counts as winning). A failed prediction may revise either the rules or the representation itself. Every real transition is appended to an immutable timeline. Inside one deliberation the agent writes code, replays the program against the full timeline, and searches inside the program only after the replay succeeds; a separate commit channel is the only operation that spends real actions, and one mismatch voids the rest of a committed plan. The authors report that the same frontier models improve from 42.83% under a generic coding-agent harness to 98.98% mean RHAE on the 25 public games. That figure is self-reported, on the public set, with a fallback to a second model on low-scoring games. The source code is not released, but 50 full trajectories and a scorer are public [1].

Retrodict

Retrodict (Ryan Brown) is the inexpensive end of the same family. It treats each game as a laboratory notebook: every frame is appended to a log, a device it takes from RGB-Agent [6], and a hypothesis is first replayed over the recorded frames in a few lines of Python. This check is the retrodiction that gives the system its name, and a hypothesis that fails it costs no game action. Each model reply must end with a plan in which every action carries the cells it expects to see afterwards; the runner executes the plan without further model calls and halts at the first expectation that fails. A short playbook file survives context resets and marks each statement as checked against the log or still assumed. A full simulator is written only as an escalation, after a level has consumed about 300 actions or two resets. Retrodict reports 99.86% mean RHAE on the public games on an official scorecard, using 7,703 actions and about 660 million tokens of a frontier model [2].

What was adopted, and how it was changed

The main departure from both systems is the division of labour. Schema and Retrodict leave perception and bookkeeping to a frontier model and enforce discipline around it. With a 27B model that division fails, so this harness computes perception and structural relations itself and asks the model only for the hypotheses that connect them.

An earlier attempt: running Retrodict unchanged on a smaller model

Before this harness was written, the unmodified Retrodict prompt and runner were ported to a smaller local model (Qwen3.8-Flash-Next) with a 32,768-token context, where the original uses about 150,000. The first runs on ls20 and ft09 scored a mean of 2.38 and failed at the level of the protocol and not of the idea. One game ended when a 24,577-token prompt plus an 8,192-token completion exceeded the context. The other ended because the model used its entire completion for reasoning and returned an empty reply with no action block. A direct-interaction baseline (the Duck agent) on the same model scored 8.94 on the 25 public games in one run on rented hardware, and 5.74 in the Kaggle notebook run reported in the results table. These results motivated the move to a 27B model and to a harness that carries more of the work.

Use of the public Schema traces

The published Schema trajectories were used in four ways, all of them offline. First, they were read as a reference when diagnosing a failed game, to compare the representation a frontier model had reached with the one Qwen had reached; for example, the reference run of bp35 describes a 6-pixel lattice in a persistent scrolling world, where Qwen had described two unrelated layouts. Second, they served as replay tests: the token simulator reproduces 417 of 417 transitions of the reference ka59 trajectory. Third, the pickup abstraction used for r11l was taken from the reference trajectory and was labelled as a hypothesis until a live Qwen transition confirmed it. Fourth, level 7 of ka59 used an action prior pruned from the reference trajectory. No reference action, coordinate or game-specific rule is placed in the model's prompt, and the first three uses shape reusable machinery only. The fourth is a direct use of a solution and is reported as such in the results.

The research question that remains is one of transfer: how much of this approach stays effective when the model is a 27B open-weight model that can be served on a single GPU, and which responsibilities must move from the model into the harness to make it work.

03Baseline and its failure modes

The baseline agent presents the current frame to the model, lets it reason, executes the action it names, and repeats. This agent fails in the same three ways on almost every game. Figure 3 shows one such run next to a run of the full system on the same game.

1. Perception. Given a 64×64 grid as text, the model spends thousands of tokens transcribing rows and still misplaces object boundaries.
2. Untested hypotheses. The model holds several explanations of an action at once (select, toggle, move) and does not design an action that would distinguish them.
3. Loss of established facts. Once the context fills, a fact established on level 1 is derived again, sometimes incorrectly, on level 4.

Without the harnessplain loop: look at the frame, reason, act

actions 0levels cleared 0

What the model said before this action

With the harnessperception, record and verification in code

actions 0levels cleared 0

What the model said before this action

Play←→2×loading

Without the harness, the model clicks seven times with no visible effect, then discovers that a click flips a tile and adopts the goal of turning every blue tile red. That is not the rule: the tiles must match the miniature pattern in the centre of each panel. After 17 actions and 31 model calls it has cleared no level, and the run ended when the accumulated context (1.48 million prompt tokens) exceeded the server limit. With the harness, the first click is a stated probe, the next three remove the remaining mismatches against the miniature, and the level clears on the fourth action. The same run then clears the remaining five levels, at actions 11, 25, 41, 62 and 75, without a single action whose outcome the model had not predicted.

A larger reasoning budget does not remove any of these failures. Additional reasoning is useful only after the model is given a correct structured description of the game.

04System design

The system separates proposing from checking. The language model proposes what the objects are, what an action does, and what completes a level. A deterministic Python harness, which contains no learned components, owns every operation that can be computed or verified exactly. No game identifier, coordinate, or level-specific answer is ever placed in the prompt or the harness.

Observation and tools

Each observation is presented losslessly as 64 rows of hexadecimal characters, one character per cell, with coordinates always given as grid[y][x]. The model can request numeric crops, connected components for every colour, and paginated retrieval of any earlier observation. Numeric colour identifiers are authoritative; rendered images are provided for geometry only, after one run spent more than 10,000 reasoning characters renaming colours it could already read as numbers. The model can also run Python in a compute tool and store reusable helper functions.

Memory

Every real action is stored as an immutable transition: the frame before, the action, the frame after and the exact set of changed cells. The model keeps persistent notes, divided into evidence and hypotheses, and a JSON scratch memory. When a note contradicts the transition record, the harness cites the contradicting transition. Context compaction keeps the current observation and the notes, removes raw exchanges oldest first, and leaves all removed content retrievable. Compact checkpoints retain relational facts (counts, bounding boxes, colour transitions, component motions, level changes) and omit duplicated cell lists, which remain in the immutable artifacts.

Executable world model and replay verification

The harness provides a scaffold for an executable model of the game with cloning, raw-grid state and common grid helpers. The language model supplies only the game-specific parts through exact source patches: parse_obs, step, render, is_goal, and optionally is_dead and legal_actions. Verification replays the model sequentially from the initial observation of each level or reset segment and never injects a later true frame to conceal drift. At a completed level the verifier checks the goal prediction and does not require the model to predict the next, unseen layout. Agreement with history is treated as evidence and not as proof of rules that have not yet been observed.

Search and guarded execution

A bounded breadth-first search runs inside the verified model and reports one of five explicit outcomes: found, exhausted, depth limit, budget limit or error. Exploratory probes are single actions. A plan may contain at most 32 actions and carries either sparse expected cells and level changes or full model predictions. The harness executes one action at a time and halts at the first mismatch, level boundary or terminal state. An inaccurate model does not prevent a probe, because probes are how the model is corrected.

Structural relations computed by the harness

Most of the engineering effort went into relations that the harness computes exactly from recorded transitions and reports to the model as hypotheses. Each was added after a specific failure in a live trace and was validated by replaying that trace.

Decision gates

Trace analysis showed that a large share of wall-clock time was spent reasoning about facts that the harness had already established. Three narrow gates replace a reasoning turn with a short commit turn when the evidence is settled. Other diagnostic turns keep the full reasoning budget.

The no-op gate would not have fired in either of the first two completed games: ft09 has no no-op among its 75 actions and sb26 has none among 124. A fourth mechanism is a structured critique that is triggered when an objective has been exhausted without progress, for example when every matching component has been removed and the level has not advanced. The model must keep verified facts, list unexplained observations, compare three to five hypotheses from at least three families (including passive or support dynamics and multi-object contact dynamics), and commit to one discriminating probe.

Code execution, logging and resumption

Model-written Python runs in disposable subprocesses with no inherited credentials, a restricted syntax tree and import list, no file or network access, and limits on CPU time, wall time and output. Each run stores an ordered event log, content-addressed copies of every request and raw response, every streamed chunk with its arrival time, all frames, and every version of the notes and the world model. An interrupted game can be resumed without repeating paid inference: the saved actions are replayed through the local engine, and every before and after frame is asserted to match.

05Experimental setup

Evaluation protocol

1. Each game starts from an empty workspace. No notes, actions, programs or facts are inherited from another game.
2. Complete requests, responses, reasoning, tool calls, observations, actions and notes are preserved.
3. The earliest causal failure in a trace is diagnosed, because a later error is often a consequence of one incorrect representation.
4. Only reusable perception, memory, verification, planning or execution machinery is added to the harness.
5. A change is credited only when a new run exercises it and improves the failed decision without breaking a previously solved game.
6. Discovery on a new game is given at least the larger of 40 actions and 1.5 times the human baseline for level 1 before the game is set aside.

06Case studies

ft09: a relation between distant regions

Without assistance the model treated each panel of tiles as a separate puzzle. The missing element was a relation between regions that are far apart on screen: the small patterned tile inside a panel is a miniature of that panel's target state. Clicking a tile toggles its colour, and the level completes when every panel matches its miniature. The harness was not changed to state this rule. It was changed to report panels in pairs, to express tile coordinates in panel space, and to list reflections and colour permutations between regions as candidate relations. After this change the model's plans took the following form.

Toggle BR tile (36,52) from blue to red to match micro swatch 0 at row2,col0. This is the last mismatch; expect level completion.

Qwen3.8-27B, third action of the completed ft09 run

The run completed all six levels in 75 actions and 26 model calls, in about eleven minutes. No action produced a result that the model had not predicted.

Observed pattern. The model did not need to be told the rule. It needed the two regions presented side by side in the same coordinates, after which it found the relation itself.

sb26: references between containers

The top row gives the order in which tokens must appear. The boxes in the middle contain slots. From level 2 onward, some slots contain a hollow token in the colour of another box, which means that reading continues inside that box. A depth-first reading of the boxes yields one sequence, and the level completes when that sequence equals the top row. A flat list of connected regions cannot express this structure, so the harness constructs the container and reference graph, in 12 to 15 ms per frame, without interpreting its edges. Given the graph, the model proposed the depth-first reading, the harness verified it against every completed level, and a finite recursive assignment solver produced the remaining placements. The run used 124 actions and 90 model calls over 44 minutes across resumed sessions. Thirteen actions produced an outcome the model had not predicted, and in each case the harness halted the plan.

Observed pattern. When the structure of a game is a graph, a list of objects is the wrong data structure, and no amount of reasoning over the list recovers it.

r11l: docking the centroid of a graph

Clicking a diamond-shaped node transfers a selection, and clicking free ground moves the selected node. Each system of connected nodes has a marker at the floor mean of its members' centres, and a level completes when every marker rests on the ring with the matching colour signature. The model initially treated the markers as decoration. After four levels the game changes its representation: the system markers begin black, and separate half-coloured octagons must be collected by passing the marker over them before docking.

Two exact-geometry errors were found through live mismatches. Collision had been approximated by a centre distance of at most four cells, which accepted the diagonal offset (4,4); the two 21-cell octagons have their corners removed and share no cell at that offset, and the game confirmed that nothing was collected. Collision is now computed from the rendered cell sets. Separately, a connector line crossing a marker added a structural colour to its signature, so signatures now accept only the item palette. The run completed all six levels in 138 actions and 145 model calls, with per-level counts of 10, 18, 39, 18, 28 and 25 actions against human baselines of 22, 33, 51, 26, 52 and 49.

Observed pattern. Both errors were in geometry that the harness computed and not in the model's reasoning. Guarded execution exposed each one with a single wasted action.

ka59: when the missing capability is executable state

In the first attempt the model made 152 calls and produced 614,648 completion tokens without clearing level 2. The decisive evidence was present in its own trace: at step 60 the selected token stayed fixed while a second token moved five lattice cells. The model described the selected token as “blocked”, continued to plan in screen coordinates, and never turned contact launching into a transition rule. The harness now owns a state model for this family of mechanics: it parses tokens on a 3-pixel lattice, models direct movement, bump launches, shoves, overshoot and countdown launches, replays recorded transitions, and plans with A*. It reproduces 417 of 417 transitions of a public reference trace and 110 of 110 transitions of the original Qwen trace.

A clean run with this controller completed all seven levels in 300 actions with no resets, using 11, 38, 33, 39, 20, 46 and 113 actions against human baselines of 28, 109, 51, 51, 33, 132 and 326. Levels 1 to 6 were planned by the simulator. On level 7 the generic A* search exceeded one million states, so the run used a 113-action prior that was pruned from a 117-action public Schema trajectory and verified from the actual entry state. This level is therefore a result about the integrated system on public data and is not evidence of generalisation to hidden games.

Observed pattern. A 27B model can describe a mechanic in prose and still be unable to plan with it. Turning the description into executable state is what made planning possible.

07Evidence from the traces

Each card below follows one failure from the live trace to the verified change, in four steps: what was observed, what caused it, what was changed in the harness, and how the change was checked. The format follows the evidence cards of the Schema write-up [1].

Evidence Abp35Reasoning about a fact the harness had already proved247 s → 5.4 s

1. Observed
   At step 3 the harness proved that two actions produce the same masked transition from the same state. The model stated this at once, then spent 36,514 reasoning characters and about four minutes deriving it again before choosing another probe.
2. Diagnosis
   The decision was correct and the cost was latency. The evidence was settled, so a full reasoning turn could add nothing.
3. Change
   A newly established conditional equivalence now forces the next call to be a short commit turn with reasoning disabled. The turn receives the exact relation and must choose a new low-risk probe. All other diagnostic turns keep the full budget.
4. Verified
   On a fresh run the forced commit took 5.37 s and 97 tokens with no reasoning tokens. The full suite passed (132 tests).

Source: PR #6, commit bcf4386, runs bp35-axis-v3 and bp35-equivalence-v4

Evidence Bar25Two objects moving in opposite directions under one actionexact motion recovered

1. Observed
   The model's notes stated that both pieces move together horizontally, and it executed a long plan from that belief.
2. Diagnosis
   Exact replay shows one action moving a 45-cell component left by 3 and a 40-cell component right by 3. The colour-delta representation only sees strips entering and leaving when a shape overlaps its old position, so it reported no rigid translation at all.
3. Change
   The harness now matches normalised connected components across frames and reports a complete translation for every object, including several objects that move differently under one action.
4. Verified
   Real steps 18, 22 and 32 replay correctly. Level 1 was then cleared at action 21 of a fresh run.

Source: PR #6, commit 10833d0

Evidence Cr11lA collision test that was wrong by one diagonal cell6 / 6 in 138 actions

1. Observed
   A capture plan placed the system marker at offset (4,4) from a pickup. The game did not collect it (step 131).
2. Diagnosis
   Collision had been approximated as a centre distance of at most four cells. The two 21-cell octagons have their corners removed and share no cell at that offset.
3. Change
   Capture checks compare the rendered cell sets. The observed successful offsets (2,4), (4,1) and (0,0) remain valid and (4,4) is rejected. Signatures accept only the item palette, after a connector line had added a structural colour to one inventory.
4. Verified
   The next captures succeeded at the predicted centroids, all sparse expectations matched, and the game returned WIN at step 138. The suite passed (156 tests).

Source: hardening ledger, section on r11l

Evidence Dka59More tokens did not substitute for executable state614,648 tokens → 7 / 7

1. Observed
   Across 152 calls the model produced 614,648 completion tokens and never cleared level 2. At step 60 the selected token stayed fixed while another token moved five lattice cells; the model called the selected token “blocked”.
2. Diagnosis
   The decisive evidence was already in the trace. The model planned in screen coordinates and never promoted contact launching to a transition rule. The missing capability was executable state and not a larger prose memory.
3. Change
   A harness-owned state model for this mechanic family: a 3-pixel lattice parser, direct movement, bump launches, shoves, overshoot and countdown launches, transition replay and A* planning, with checked execution of at most 16 actions per batch.
4. Verified
   The simulator reproduces 417 of 417 reference transitions and 110 of 110 original Qwen transitions. A clean run then won all seven levels in 300 actions. Level 7 used a disclosed prior from a public trace.

Source: hardening ledger, section on ka59; PR #6, commit 906bc96

Evidence Edc22A checkpoint too large for the context on its own170,280 → 97,912 characters

1. Observed
   The run failed at action 17 with an oversized request, although every raw exchange had already been evicted.
2. Diagnosis
   The remaining checkpoint was 170,280 characters. Exact cell arrays were duplicated in the verified ledger, the recent evidence and the last feedback, and the full structural graph was repeated as well. A second compaction had nothing left to evict.
3. Change
   Compact checkpoints keep the relational facts a decision needs (counts, bounding boxes, colour transitions, component motions, level changes) and omit the duplicated arrays, which stay available in the immutable artifacts.
4. Verified
   The same 17-action trace now produces a 97,912-character checkpoint: 96,416 of 131,072 tokens with the full 32,768-token diagnostic reserve. The game resumes with its actions, notes and board state intact.

Source: PR #6, commit e18ce1a

08Results

Six of the 25 public games are complete with the maximum local score of 100. Each entry below opens to a replay of the actual run, recorded action by action, together with an account of where the model alone got stuck, what the harness supplies for that game family, and why that is sufficient. A replay loads only when its entry is opened.

6 / 6ft09Single run from empty memory; no mispredicted action75 actions · human 208

actions 0levels cleared 0

Reason logged with this action

Play←→2×loading

Where the model alone got stuck

A flat list of coloured regions hides the fact that each panel contains a worked example of its own target. The model treated the panels as separate puzzles, and on later levels it lost track of palettes that change from level to level and of glyphs that cycle through several colours.

What the harness supplies

• A table of tile positions for every repeated lattice, in panel coordinates and not pixel coordinates
• Ordered runs of solid swatches, so that a miniature can be read as a sequence
• Observed forward and reverse palette edges: which colour a click turns into which
• A guard that stops an empty marker with a uniform surround from being read as a tile
• Replay certification of a complete glyph plan, followed by deterministic execution

Why that is enough

With panel coordinates and the miniature side by side, the comparison between them is a lookup that the model performs reliably. The plan for a level is then certified by replay before any action is spent, which is why no action in this run had an outcome the model had not predicted.

Actions per level

L1

434

L2

127

L3

2314

L4

2816

L5

6521

L6

3713

human baseline this system   26 model calls in total

Duck baseline on this game [8]: 2 of 6 levels.

8 / 8sb26Completed across resumed sessions; 13 mispredicted actions124 actions · human 213

actions 0levels cleared 0

Reason logged with this action

Play←→2×loading

Where the model alone got stuck

Flat objects cannot express containers nested inside containers, hollow tokens that refer to another box, a reading order that recurses or cycles, or a stable assignment of tokens to slots. The model described individual boxes correctly and could not combine them.

What the harness supplies

• A container and reference graph: boxes contain slots, slots contain solid tokens or hollow references
• A finite recursive assignment solver over that graph
• Verification of the goal predicate against real submissions
• Decision guidance that points at the contradicting transition when notes and record disagree
• Certified plan compilation and automatic execution, including cyclic traversal

Why that is enough

The graph gives the model one object to reason about in place of dozens of rectangles. It proposed the depth-first reading itself; the harness checked that reading against every completed level and then solved the assignment, so later levels were mostly bookkeeping.

Every one of the 13 mispredicted actions halted its plan at once, which is visible in the replay as a highlighted step.

Actions per level

L1

189

L2

2815

L3

1815

L4

1915

L5

3117

L6

2319

L7

5817

L8

1817

human baseline this system   90 model calls in total

Duck baseline on this game [8]: 1 of 8 levels.

6 / 6r11lEvery level below its human baseline; pickup representation informed by a public reference trace138 actions · human 233

actions 0levels cleared 0

Reason logged with this action

Play←→2×loading

Where the model alone got stuck

The model treated the markers at the centre of each graph as decoration, so it never formed the idea that a marker is the mean position of its nodes. Later, the harness itself approximated octagon collisions with bounding boxes, and a connector line that crossed a marker contaminated its colour signature.

What the harness supplies

• A graph parser that recovers systems, nodes and the floor-mean marker of each system
• Ordered pickup signatures restricted to the item palette
• Exact octagon overlap computed from rendered cells
• A planner for capture and docking that moves whole footprints
• Verified execution across several systems, protecting docks that are already complete

Why that is enough

Once the marker is defined as a computed quantity, moving it to a target is a geometry problem that the planner solves exactly. The model's role reduces to choosing which system to complete next and to designing the probe that confirmed how pickups are collected.

Actions per level

L1

2210

L2

3318

L3

5139

L4

2618

L5

5228

L6

4925

human baseline this system   145 model calls in total

Duck baseline on this game [8]: 1 of 6 levels.

7 / 7ka59Levels 1 to 6 planned by the simulator; level 7 used a disclosed prior from a public trace300 actions · human 730

actions 0levels cleared 0

Reason logged with this action

Play←→2×loading

Where the model alone got stuck

The model represented boxes on the screen and not objects on a lattice, read the digits on tokens as labels when they are launch distances, and kept retrying direct movement after contact had launched a different object. In 152 calls and 614,648 completion tokens it never cleared level 2.

What the harness supplies

• A parser for tokens, frames, walls and countdown gauges on a 3-pixel lattice
• An object and contact simulator: direct moves, bump launches, shoves, overshoot, ranged countdown launches
• Replay of the simulator against recorded transitions (417 of 417 reference and 110 of 110 original)
• An A* planner and a compiler from plans to sparse per-action expectations
• Checked execution in batches of at most 16 actions

Why that is enough

The mechanics are deterministic, so once they are executable the game becomes a search problem. This run needed only two model calls; the rest was planning and checked execution.

On level 7 the generic search exceeded one million states. The 113 actions shown for that level come from a prior pruned from a public Schema trajectory and verified from the real entry state, so that level is not evidence of generalisation.

Actions per level

L1

2811

L2

10938

L3

5133

L4

5139

L5

3320

L6

13246

L7

326113

human baseline this system   2 model calls in total

Duck baseline on this game [8]: 0 of 7 levels.

8 / 8lp85Completed across resumed sessions; the final 21 actions needed no model call162 actions · human 388

actions 0levels cleared 0

Reason logged with this action

Play←→2×loading

Where the model alone got stuck

Level 2 couples several tokens on interlocking tracks, and the model spent a full ten-action lap on one row and reversed its only useful move. Level 3 replaces the rectangular layout with two overlapping sparse cycles, which a rectangular parser split into unrelated columns. On level 8 it transcribed visible squares repeatedly and did not see that one operation appears at three scales.

What the harness supplies

• Exact tile permutations learned from observed transitions, for rows, perimeters, pinwheels and sparse cycles
• A sparse lattice parser: dominant aligned phase, eight-neighbour tile graph, bounded cycle enumeration
• Tracks recovered as a diagonal lead-in followed by a straight stem, with entry registers and a shared conveyor
• Joint search over all token positions, with tokens tracked by halo colour
• Plans compiled to checked actions: every token coordinate is verified after every click

Why that is enough

Each control is a permutation of tile positions, and a permutation can be learned exactly from one or two observations. After that, reaching the targets is a joint shortest-path search. On level 8 the controller found a 20-action plan after six probes and used 27 actions in total against a human reference of 159.

Actions per level

L1

177

L2

3841

L3

3123

L4

1617

L5

4114

L6

6020

L7

2613

L8

15927

human baseline this system   147 model calls in total

Duck baseline on this game [8]: 4 of 8 levels.

6 / 6tr87Completed with no resets and no mispredicted action; the final 23 actions needed no model call189 actions · human 414

actions 0levels cleared 0

Reason logged with this action

Play←→2×loading

Where the model alone got stuck

The game is about symbols and a dictionary that maps one symbol to another. The model read editable glyph identities as rotations and reflections of one another, transcribed the finite symbol wheels again and again, took editable dictionary boxes for fixed examples, and could not compose the final chain of two mappings (10 to 7 to 11).

What the harness supplies

• Neutral identities for framed glyphs, so that a symbol is a name and not a picture to be rotated
• Persistent edges for each finite symbol wheel and for the cursor, learned once from observed transitions
• A guard that stops the agent from undoing its own last edit
• A parser for framed words and a direct constraint solver for the editable dictionary
• Composition of chained finite transducers, with checked execution one step at a time

Why that is enough

Once glyphs are names and each wheel is a known cycle, a level is a small constraint problem: which dictionary entries must change so that the input word maps to the target word. The solver answers that exactly, including when two mappings have to be chained, and the plan is executed with a frame check after every action.

Actions per level

L1

5432

L2

5838

L3

4041

L4

4536

L5

7118

L6

14624

human baseline this system   65 model calls in total

Duck baseline on this game [8]: 1 of 6 levels.

All 25 public games

Six games are complete, ten are in progress (the furthest are su15 with 6 of 9 levels and re86 with 6 of 8) and nine have not been attempted. Scores for games in progress are the local scorecard values so far and will change. The four-character identifiers are the official ones. The longer names are unofficial backronyms from a community list; I changed four of them (g50t, su15, re86, tr87) after checking the mechanics in the public Schema notes [1]. Select a name to read what the game asks for. For every level the table gives the human reference actions next to the actions this system used. The last line of each game is a reference point: the levels, actions and score of the Duck agent [8], the open-source winner of the first milestone, in my Kaggle notebook run of 14 September 2026. That run cleared 32 of 183 levels across the 25 games, cleared at least one level in 19 games, completed no game, and had a mean score of 5.74. On the six games completed here it cleared 9 of 41 levels.

09Negative results

An all-or-nothing verifier provides no gradient

The first verifier asked the model for a complete simulator and counted a transition as correct only if all 4,096 cells matched. One wrong cell scored the same as a crash, and because the simulated state was rolled forward, one early error invalidated every later transition. The verifier now reports separately whether the code ran, how many transitions are exact, what fraction of cells is correct, and which transition fails first. New code replaces old code only if it improves on this backtest.

Persistent memory can preserve an incorrect belief

In one resumed sb26 session the model's notes still stated that a click erases and repaints a token, long after the transition record showed the token moving as a block. Notes are now presented with their age and with the harness's own record of what each action caused. After a structural upgrade, a single short turn rewrites notes that the new relations contradict; on dc22 this took 33.6 seconds and 805 tokens and removed a false claim that two glyphs had disappeared.

A checkpoint can overflow the context on its own

On dc22 the run failed at action 17 even though every raw exchange had already been evicted. The remaining checkpoint was 170,280 characters, because exact cell arrays were duplicated in the ledger, the recent evidence and the last feedback. Compact checkpoints reduced the same trace to 97,912 characters, which fits in 96,416 of 131,072 tokens with the full diagnostic reserve.

More reasoning about settled evidence is not useful reasoning

The clearest example is the 325-second reply on ar25, in which the model selected the correct next probe near the start and then spent 15,919 tokens constructing an expectation for it. The probe changed 487 cells and was informative. The failure was latency and not judgement, which is why the decision gates are narrow.

10Limitations

• All results are on public games that were inspected during development. They constitute a development set.
• Several harness components were written after studying a failure on a specific game, and two of them (the token simulator and the pickup representation) were informed by public Schema traces. No game identifier or solution is encoded, but the risk that the harness is fitted to the public mechanics is real and can only be measured on games that were not studied.
• The sb26, ka59 and lp85 results were obtained across resumed sessions, and level 7 of ka59 used a public-trace prior.
• The model is served on an H200. The competition requires an offline submission, and the model and harness have not yet been packaged for it.
• Nine of the 25 public games have not been attempted.

11Planned direction: test-time communication

Park, Kontonis, Garg, Krishnamurthy and Papailiopoulos [9] compare two ways of spending the same number of agents on a task. In best-of-N, the agents work in isolation and the best result is kept, so every agent has to make every discovery by itself. In team-of-N, the agents share an append-only channel, so a discovery has to be made only once. On ARC-AGI-3 they report that a team of k communicating agents matches the success rate of about 4k independent agents, that the advantage grows with k (a team of five matches best-of-33), and that a team can reliably solve a game that no single agent solved. They also state the conditions: communication costs tokens and time, and independent agents can do better when compute is limited or when there is no clear measure of progress. It helps most when the task needs several successive discoveries, partial progress transfers between attempts, and a dense verifier exists.

What the paper's protocol actually is

• Homogeneous agents. Every agent has the same model, tools, task instruction and communication prompt. There are no assigned roles such as player, critic or planner.
• Private work, shared findings. Each agent keeps a private context and scratch directory. All agents share append-only records of findings, disconfirmations, scores and coordination.
• Approach slots. Agents claim distinct approach slots at the start to avoid early herding, publish concise reproducible evidence, and adopt a peer's approach only after a measured improvement or a genuine block.
• Separate game sessions. On ARC-AGI-3 each agent has its own authenticated game session and its own per-level action budget. An action or a level clear by one agent does not advance a peer's game. Only discoveries are shared.
• Synchronisation barrier. Every half-level action budget, an agent that is ahead of the slowest live teammate cannot spend another game action until its peers catch up or leave the live set. It can still read files, run code and exchange notes.
• Pooled metric. A level counts as solved if any member clears it, and a game counts as solved if any member clears all its levels. The RHAE analysis reports the best and the average individual member separately and does not merge the agents' actions into one trajectory.

The paper therefore supports several communicating sessions directly. It does not validate several agents writing to one board, and its pooled accounting cannot be assumed to match how a competition submission is scored.

Why it fits this harness

The paper's conditions describe this setting. A game is a chain of discoveries (what the objects are, what each action does, what the goal is), a discovery made on one level carries to the next, and replay against the transition record is a dense and exact verifier. The mechanism I intend to borrow is narrow: independent search followed by rapid adoption of verified discoveries. Agreement or commentary between agents that is not verified adds cost and no capability.

When the 27B model finds the right representation it completes whole games. Its recurring failures are failures of the search process: it stays committed to one interpretation for too long, it recounts geometry by hand that code could compute, it knows the local mechanics and lacks the global objective, it describes a simulator without writing or testing one, and it loses or reinterprets a fact it had already established. Four independent contexts can search different representations without disturbing one another's working state, and the verifier lets one good partial discovery cross between contexts without importing unsupported conclusions.

All agents call the same Qwen3.8-27B server, so no additional copy of the weights is loaded. The cost is four KV caches, more generated tokens, contention for decoding throughput and, in the paper-faithful variant, four times the environment actions. The development server holds well over four full 131,072-token sequences; four is its scheduler cap. Whether four contexts fit the throughput and the nine-hour limit of the Kaggle GPU has to be measured there.

Two variants

Paper-faithful: four identical agents, four game sessions. Each agent has its own context, scratch space, game instance and action budget, and all publish to the shared logs. One agent may discover that an action translates an arm while another discovers that a different action rotates it; the others reproduce or build on those facts in their own sessions. This is the closest reproduction and the strongest source of independent experimental evidence. It multiplies environment usage by four, and it is a development technique until it is known whether the hidden evaluation permits several scored sessions and how it would aggregate them.

Submission adaptation: four identical agents, one action stream. All four receive the same prompt, tools, current observation, verified history and shared log, and all may inspect frames, write code, patch candidate models, backtest, search and propose experiments. Only one real action or batch is committed at a time. There is no permanent player: the harness owns the action endpoint, any agent may submit a proposal with its prediction and evidence, and the harness accepts one, executes it, records the transition and broadcasts the result. This serialises a contested resource; it does not assign a cognitive role. It keeps one scored action stream and avoids stale-state races. It departs from the paper in that the other three agents cannot gather distinct real observations in parallel. They can still mine the shared history, test competing representations and, once an executable world model exists, search private simulated branches without spending real actions.

For the single action stream, play proceeds in synchronised discovery rounds:

freeze the current observation and the verified history
-> four agents independently inspect, test and search
-> agents append concise discoveries and action proposals
-> the harness verifies what can be checked
-> exactly one proposed real action or batch is committed
-> the transition and the verification outcome enter the shared log
-> every private context continues from the common new state

What is shared, and who decides what is true

Agents receive evidence and not one another's reasoning: frames and component tables, transitions, harness-derived changes and object tracks, verified notes, rejected hypotheses, the current world model with its backtest report, and the remaining budgets. A contribution names a representation, hypotheses, a discriminating experiment with its prediction, a simulator patch or a candidate plan, and the evidence it used. The verifier, and no model, owns the truth: a contribution enters shared memory only after replay (for a transition rule), an exact structural check (for a claim about a frame), predicted against observed (for a probe), search replay (for a plan) and a contradiction check against all retained evidence. The channel is append-only and typed, because silently replacing a belief would destroy the evidence trail, and raw monologues stay in per-agent traces.

{
  "kind": "discovery | disconfirmation | model_patch | plan | request",
  "level": 6,
  "state_id": "...",
  "author": "agent-0 | agent-1 | agent-2 | agent-3",
  "claim": "...",
  "evidence": ["transition ids"],
  "prediction": "...",
  "verification": "pending | passed | failed",
  "artifact": "optional model or plan reference"
}

Diversity does not come from different prompts, which stay identical as in the paper. It comes from atomically claimed approach slots, private scratch space, distinct working hypotheses, and the instruction to keep one meaningful variation even after adopting a peer's better result. The design fails if all four agents repeat one interpretation and only critique its wording.

Revised live design: an asynchronous three-context event bus

A negative result first. On 21 September a probe on the development server rejected the simplest version, two independent contexts whose answers are merged at the end. Both contexts spent roughly the same first 8,000 reasoning tokens reconstructing the same cross geometry on level 6 of a game and arriving at the same contradiction about coverage. Merging two duplicate answers is best-of-2. It is not the communication effect.

The next probe therefore uses three Qwen3.8-27B contexts on one board, and for this probe they do have distinct responsibilities, which departs from the homogeneous design above:

• Master. Owns strategic continuity, hypotheses about the objective and the decision criteria, and will eventually be the only context allowed to take real actions. It must not spend its context transcribing geometry by hand.
• World modeler. Owns the exact representation of the board, the causal mechanics, executable helpers and models, backtests and discriminating experiments.
• Communication coordinator. Receives completed reasoning, tool calls, tool results and structured reports from both workers. It removes duplicates, keeps evidence references, flags conflicts and assigns distinct next questions. It never acts and should not try to solve the board itself.

Communication is asynchronous at inference boundaries. A transformer request cannot absorb new context while it is generating, so the harness appends an event to the shared bus as soon as a context finishes a reasoning or tool turn, and every context receives all unseen events from the others before its next inference call. This is the same steering point at which a tool-using agent receives a message between two tool calls. Every event carries a sequence number, an author, a kind, a board state version, a timestamp and a payload. A report about an older board version remains useful as historical evidence and cannot authorise an action against a newer state.

What is shared. Completed raw reasoning goes to the coordinator, because it shows which paths were already explored, which alternatives were discarded, which leaps were unsupported, and which insights never reached the final report. Tool calls and results are shared at once, so that a second worker does not repeat an exact measurement or a program that already failed. The coordinator turns these into a compact handoff: new discoveries grounded in evidence, contradictions with their provenance, open questions, separate assignments for the master and the world modeler, and the references needed to audit each claim. Workers may also see newly completed raw events before the next compaction, which allows fast steering between tool calls. The compact handoffs are the durable shared memory; raw monologues are diagnostic evidence and not permanent prompt state.

Context policy. The event file is append-only and lossless, and each prompt uses a bounded first-in, first-out inbox:

1. Before a request, collect all unseen events.
2. Reserve the 32,768-token completion window and the current tool schemas.
3. Ask the provider's own fits() check whether the request fits.
4. If it does not, remove the oldest raw event and retry until the newest suffix fits.
5. Record the included sequence range and the number of omitted events in the trace.

Old reasoning therefore leaves the active context without being deleted. Persistent notes and coordinator handoffs keep the useful conclusions, the full log remains available for audit or targeted retrieval, and at the next collaboration round each role starts from a clean role-specific checkpoint.

Test criterion. The architecture is useful only if the trace shows causal transfer: (1) one worker discovers information that was absent from the other worker's previous turn; (2) the receiving worker explicitly changes or narrows its next investigation; (3) duplicated reasoning decreases after the event arrives; (4) the coordinator preserves the discovery without promoting speculation; and (5) the final master plan contains a verified improvement over either worker's initial report alone. The probe runs with reasoning enabled and the full 32,768-token reply ceiling, with no smaller cap on diagnostics or handoffs.

How it composes with the existing harness

Retrodict [2] contributes explicit predictions, expectations, contradiction handling and a compact evidence log. Schema [1] contributes executable models, replay backtests, repair and planning inside the inferred model. Test-time communication lets separate contexts discover and improve those artifacts while sharing only verified progress. It is not a teacher: every agent is the same base model, sees only the current game's evidence, and receives no released solutions or privileged action sequences.

Experiment required before adoption

Arm C measures whether the paper's effect reproduces with a 27B model. Arm D measures whether the effect survives the single action stream that a submission is likely to need. The measures are levels and games completed, actions and RHAE, wall time and tokens, actions to the first correct representation, the number of proposed discoveries that pass verification, the rate of repeated hypotheses and contradictions, improvement in the model backtest, peak KV-cache memory and decoding throughput, and whether a communicated discovery is actually adopted and reused. Adoption requires an improvement in completion or action efficiency across several mechanic families within the Kaggle runtime limits, and not a higher success rate on one public game.

Open questions: whether all four agents should run continuously or only while a mechanic is unresolved; how the single-stream harness should choose among simultaneous proposals without becoming a learned central orchestrator; how long a synchronised round may last before an action must be taken; which discoveries transfer across levels and which are local to one; and when the harness should abandon communication and take the best verified action available.

12Future work

• Run the four-arm test-time communication experiment described above (one agent, best-of-4, team-of-4 with separate sessions, team-of-4 with one action stream) on games with known stalls.
• Complete the sweep over all 25 public games, recording the earliest causal failure on each.
• Freeze the harness and repeat the completed games from empty memory several times to measure reliability, with ft09 and sb26 as regression sentinels.
• Report time, tokens, real actions and the number of harness interventions as separate quantities.
• Package the model and harness for the offline Kaggle environment and measure how many games can run concurrently within the available KV cache.
• Train on intermediate decisions (object roles, discriminating probes, repair location, stopping) and not only on final action sequences.

13Cite this page

@misc{kancharla2026verified,
  title        = {Verified World Models for Interactive Reasoning in ARC-AGI-3},
  author       = {Kancharla, Ritwika},
  year         = {2026},
  howpublished = {\url{https://ritwikareddykancharla.github.io/projects/arc-agi-3.html}},
  note         = {Work in progress}
}

14References

1. G. Zeng, J. Wang, W. Ma, S. Yin, C. Wang, S. Liu, A. Kanazawa, W. Ni, X. Li, A. Zanette and H. Feng. Schema. Project page, 2026. schema-harness.github.io. Trajectories and scorer: schema-harness/arc-agi-3-schema-traces.
2. R. Brown. Retrodict. Source code, 2026. github.com/ryanbbrown/Retrodict. Write-up: How I accidentally got the top score. Official scorecard: arcprize.org/scorecards/9c403765.
3. H. Tang, D. Key and K. Ellis. WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment. 2024. arXiv:2402.12275.
4. Tycho. 2026. arXiv:2607.28287.
5. Rodionov. Executable world models for ARC-AGI-3, 2026, arXiv:2605.05138, and the accompanying ablation study, arXiv:2607.15439, which reports that an executable model without replay verification can perform worse than plain text.
6. RGB-Agent (alexisfox7). Source code. github.com/alexisfox7/RGB-Agent. The append-only log and the action queue in Retrodict follow this agent.
7. ARC Prize Foundation. ARC-AGI-3 methodology and the RHAE metric. docs.arcprize.org/methodology.
8. Tufa Labs. The Duck, first place in ARC-AGI-3 Milestone Prize #1, open-sourced as a Kaggle notebook. Announcement: arcprize.org/blog/arc-prize-2026-milestone-1.
9. J. Park, V. Kontonis, S. Garg, A. Krishnamurthy and D. Papailiopoulos. Scaling Discovery through Test-Time Communication. 2026. arXiv:2609.21032.