Terminal policy iteration

Choose between moves by actually playing each of them forward many times and seeing which one leads to the longest, highest-scoring games — then change the reference policy's mind only when the evidence is overwhelming.

Why lifetime, and why this family exists

In this game, almost the whole score is survival. Across 64 games of the reference policy, the correlation between final score and moves survived was r = 0.9995, and 94.3% of all points came from the flat 17,000-point bonus for surviving a row rise — 5.7% came from chain waves and none at all from board clears (finding 01, exploratory, development tier, 64 paired games). Playing beautifully is not the objective. Not dying is the objective, and points arrive as a by-product at a near-constant rate.

That reframing exposes a structural problem with the reference search. It looks four of the player's moves ahead. The board rises every five moves. So the reference can barely see one rise boundary, while the thing that actually ends its games is a shortfall that accumulates over dozens of them: it clears about 1.973 discs per move where 2.400 are needed to break even, and reveals about 1.090 covered discs per move where 1.400 are needed. Its mean lifetime is 94.06 moves; a million-point average needs roughly 294. A four-move search cannot see a 40-cycle deficit, no matter how carefully it evaluates the board in front of it.

This family's answer is to stop evaluating boards and start measuring outcomes. Do not ask a hand-tuned function how promising a position looks. Play it.

How it works, step by step

1. At a position, list every legal column — not just the ones a policy likes. This is the deliberate cure for the sibling trap, where a model is trained only on moves that were actually played and then asked to rank moves that never were.
2. Generate a fixed set of imagined futures: which discs will be dealt, which hidden numbers will be revealed. Crucially, the same set is used for every column, so no column can win by being dealt luckier cards.
3. For each column, force that move, then let a simple public policy — usually the one-move-ahead or two-move-ahead fair search — play on for a fixed horizon: 25, 50, 75, 100 or 200 moves depending on the experiment. Record the score, the moves survived, and the clear and reveal rates.
4. Average across the futures. Now every column has a measured outcome instead of a heuristic estimate.
5. Do not simply take the best. Compare the winner with what the reference policy would have played, and switch only if a one-sided confidence bound on the paired difference clears a threshold that was frozen before the data was read. Otherwise keep the reference's move.

Step 5 is the "policy iteration" half of the name and it is where most of the family's discipline lives. One step of improvement over a known-decent policy is a classical, well-understood operation; the danger is that noisy measurement makes a worse move look better, and a rule that switches on noise is worse than no rule at all.

What happened, in plain English

Every candidate in this family was retired, and the family's most valuable output turned out to be a dataset rather than a policy.

The recurring shape of the failure is easy to state. When the measurement is cheap enough to run, it is too noisy to beat the reference; when it is faithful enough to trust, it is far too slow to play a game. The most faithful design of all — comparing candidate moves by playing them to the actual end of the game — was measured at roughly hundreds of seconds per move and was paused before it produced a policy result. The cheapest version, using fifteen imagined futures and fifty-move continuations, did improve every one of its four paired games, and still failed: its frozen gate demanded a 1.20× gain in both score and lifetime and it produced about 1.07× and 1.03×, while its rate of uncovering hidden discs actually fell.

What the family did produce is a locked panel of 477 positions taken from eight complete games, with every legal column at every position labelled by long continuations under shared imagined futures. That corpus is the closest thing in this repository to the successor-closed data the sibling ranking problem calls for, and it has since been used to test three quite different rankers offline, without opening a single new game. All three lost to the reference search. One was statistically indistinguishable from simply keeping the reference: it changed its mind at 11 of 477 positions, six of those changes helped, and the confidence bounds on its advantage were negative.

That negative is worth reading precisely, because it is easy to over-claim in either direction. It does not show that long continuations are the wrong signal. It shows that these particular measurements, at these budgets, on this panel, could not separate the reference's move from a better one reliably enough to act.

The technical record

Approach	Status and evidence	Recorded outcome
Terminal rollout	Runtime-paused — task-record only	A faithful full-terminal design projected roughly hundreds of seconds per move
Public survival rollout	Rejected — task-record only	31 aligned 100-move continuations were worse than fair D1 on fitting games
Terminal policy iteration	Rejected — task-record only	255 aligned 200-move continuations: the gain was real but missed the frozen gates
Public rollout policy iteration	Rejected — ledger-recorded	1.0697× score and 1.0256× moves against a 1.20× gate
Deployment panel	Mixed — see the page	The 477-root corpus, one completed diagnostic, and two rejected rankers
Public regenerative B0	Rejected — ledger-recorded	Overrode 11 of 477 roots; every gate but one failed

A caveat about the family's name. The lifetime framing above is the motivation, and two of these programs did require a gain in survived moves as well as score. But the locked 477-root panel labels siblings by score return over a truncated continuation, and the panel rankers were judged on how well they reproduce that label. Given how tightly score tracks lifetime over whole games, these are closely related quantities — but a truncated continuation return is not the same thing as a completed game's lifetime, and no experiment in this family has ranked siblings by measured lifetime alone.

A caveat about what a panel result is. Everything measured on the 477-root corpus improves only the root action; every later move reverts to the continuation policy. These are offline root-panel proxies, not complete-game results, and the ledger says so explicitly.

Sources: the terminal-policy-iteration table in the experiment index; the ledger entries "Corrected-17k public rollout policy iteration", "Full-panel conservative-policy preflight", "Martingale-dual B0 ranking audit" and "Public regenerative policy-iteration B0" in the experiment history; and the summary in research status.

What this taught us, and what is still open

• Measuring outcomes is not automatically better than evaluating boards. A long continuation replaces one source of error, a hand-tuned leaf, with another, sampling noise, and the second is not obviously smaller.
• Conservative overrides need overwhelming evidence, not merely positive evidence. Two independent experiments here produced positive mean differences whose confidence bounds were negative, and both were correctly rejected. A wrong override costs more than a right one gains.
• A policy that almost never acts has not solved anything. Coverage is a first-class metric in this family for a reason: an override that fires at 11 of 477 positions cannot move a mean, whichever way it points.
• Still open: a comparison that is cheap by design. The strategy catalog calls multi-cycle comparison one of the more informative open directions and says the existing implementation is neither fast nor robust enough. It also warns that the one striking positive pilot in this line of work — a 25-move rollout veto, which lives in the D4 and long-outcome family — must not be turned into a tuning target.