Judging a move by playing the game out

Instead of looking a fixed few moves ahead and then guessing what the board is worth, play the game out. For each column, simulate the next eight, sixteen or twenty-four moves with a fast policy, average the results, and take the column that did best. It finished behind the fast policy it was built on.

The intuition

A look-ahead search needs an opinion about the boards at the bottom of its tree, and that opinion is hand-written and imperfect. A rollout sidesteps it: instead of evaluating a position, finish the game from it — or at least play a long stretch — with some quick policy, and use the actual outcome.

This is policy improvement in its most literal form. If you have a fast policy and you rank the root moves by "what happens if I play this column and then follow the fast policy", the resulting policy is, in theory, at least as good as the fast one. Two practical things break that guarantee:

1. You are estimating, not computing. Each column's value is an average over a handful of imagined futures, and Drop7 outcomes are heavy-tailed. With too few futures, the ranking is noise.
2. The continuation is the ceiling. A rollout tells you what happens if you keep playing like the fast policy. If the fast policy squanders the structure your first move created, the rollout will report that squandering as the move's value.

The experiment controls the first problem carefully — every column is evaluated on the same seven imagined futures, so the comparison is paired — and it ran straight into the second.

How it works, step by step

1. At the root, take every legal column.
2. Build seven deterministic scenario tapes. Reveal draws and future visible discs live in separate event-indexed domains, and the first future disc is exactly stratified over all seven values, so the seven tapes cover the immediate next disc exactly once each.
3. For each column, replay all seven tapes: the first move is the column being tested, and every later move is chosen by the fair one-move policy from the public board, the visible next disc and the rise clock only — the continuation policy never sees the tape.
4. Continue for the configured horizon: 8, 16 or 24 moves. At most 9,408 simulated transitions per decision.
5. Average each column's seven returns and play the best.

The frozen fitting grid crossed the three horizons with two settings of a tail-weighting term on the fair evaluator (off, or 0.25), on 12 complete games, using plain mean returns.

What happened, in plain English

The rollout policy lost to the very policy it used as its continuation. The fair one-move policy averaged 72,526 points and 53.75 moves; the best rollout configuration — a sixteen-move horizon with no tail weighting — averaged 64,305 points and 45.5 moves. It lost all twelve leave-one-out score comparisons and all twelve clear-throughput comparisons. It cleared 1.51 numbered discs per move against the fair policy's 1.69.

That is unusually clean for a negative result. The sixteen-move horizon was also selected in eleven of twelve leave-one-out fits, so the loss is not an accident of picking the wrong grid point — the whole grid was weak. The candidate missed both frozen thresholds (250,000 points, and 1.05 times the clear rate), so the one-shot held-out range was never opened.

The most useful thing this rules out is narrow and precise: short mean-return rollouts whose continuation is the fair one-move policy. It says nothing about rollouts with a stronger continuation policy or a longer-lived value model — the ledger states that limitation itself.

The technical record

Source. approaches/fair-expectimax/rollout-improvement/fair-d1-rollout-improvement.cpp. Experiment index: rejected — ledger-recorded, "it failed the fitting gate." Ledger section "Public fair-D1 rollout-improvement pilot".

Fitting, 12 complete games on 0x3df00000...00b:

Arm	Mean score	Mean moves	Numbered clears / move
Fair one-move policy (benchmark)	72,526.17	53.75	1.69147
Best rollout (horizon 16, tail scale 0)	64,304.83	45.5	1.51282

Lost 12 of 12 leave-one-out score comparisons and 12 of 12 throughput comparisons. Horizon 16 was selected in 11 of 12 leave-one-out fits. Frozen gates: 250,000-point fitting threshold and a 1.05× clear-throughput threshold — both missed, so 0x3df10000...00f was never read.

Scoring mode. The section carries no scoring label; audit 03 classifies both means as historical 7,000-point Sequence scoring from the score identity. Score and survival regressed together, so rescoring widens the gap rather than closing it.

Interface. Defining DROP7_FAIR_D1_ROLLOUT_IMPROVEMENT_LIBRARY embeds the chooser without its standalone entry point; the reusable decision object retains each legal column's mean and its seven aligned scenario returns.

Verification. Optimised -Werror, library-mode -Werror, and ASan/UBSan self-tests passed exact determinism, reflection, metadata blindness, tape alignment and stratification, reveal/visible-domain independence, legality, and the resource bound. Peak resident memory 2,162,688 bytes. Artifact SHA-256 82b6fe9c78de486ad550dac795f76ff162799846f53a80a76177114ea0b91428.

Related evidence elsewhere in the repository. Longer and more expensive rollout variants outside this family reached the same conclusion from the other direction: a 31-continuation, hundred-move public rollout was worse than the fair one-move policy on fitting games (task-record only), and a 255- continuation version with strict confidence bounds produced a real but insufficient gain that missed its frozen gates (task-record only). Both are in the experiment index under terminal policy iteration.

What this taught us, and what is still open

Rollouts inherit their continuation's blind spots. Ranking columns by "what happens if I keep playing like this" cannot discover a plan the continuation policy would not follow, and in Drop7 the whole difficulty is preparing structure several moves before it pays. A one-move continuation dismantles that preparation as fast as the root move creates it.

The negative result is also a cost lesson. Seven futures per column at a sixteen-move horizon is already thousands of simulated transitions per decision, and it bought a policy weaker than the four-move search that costs a comparable amount. Any future rollout design in this repository has to argue not just that it is better than its continuation, but that it is better than spending the same computation on look-ahead.

Still open, and explicitly not rejected by this run: rollouts with a stronger continuation policy; rollouts whose returns feed a learned value model rather than a direct ranking; and rollouts used as a veto on a small set of near-tied root moves rather than as the ranking itself. That last shape did produce one promising pilot elsewhere in the repository — a costly 25-move rollout allowed to veto the four-move search passed its quality gates and was then stopped for missing its runtime limit by a wide margin (runtime-paused — ledger-recorded, see the experiment index).

Sources

• fair-d1-rollout-improvement.cpp — a deliberately small, public-information rollout pilot; every legal root action is evaluated on the same seven deterministic stratified scenario tapes, and every later action is chosen by the fair one-move policy without access to the tape.