Testing the benchmark instead of the policy

Grade nine policies on a fixed set of 128 saved positions instead of on whole games, then check whether that ranking agrees with how those policies actually score over complete games. It does not, and this approach says so with a number.

Why anyone wanted this

Whole games are expensive and noisy. Scores in this game are heavy-tailed, so telling two policies apart takes dozens of complete games, and each game of the reference search costs seconds of compute. A saved-position benchmark is the obvious economy: freeze 128 interesting positions, let every policy play a short window from each, average the results, and compare. Cheap, repeatable, and the same positions every time.

The repository built exactly that, and the design document that built it ended with a gate on itself: until the benchmark is shown to predict whole-game strength, it is a diagnostic and cannot be cited as evidence of policy strength. That check had never been run. This approach runs it.

The intuition: what a nine-move window can and cannot vary

Consider what a policy is actually being scored on in a nine-move window. In this game mode almost the entire score is a flat 17,000 points every fifth drop, when the board rises. Nine moves contains one or two of those rises — and which of the two it contains is decided by the rise clock the position started with, not by anything the policy does. Unless a policy dies inside the window, every policy collects exactly the same rise bonuses.

So the measurement decomposes like this, over the six real search policies:

• the rise bonus is 92% of the scenario score, and across all six policies it spans 492 points;
• chain reactions are the rest, and span 243 points;
• board clears — the 70,000-point bonus — contributed zero, because in 7,004 scenario plays the board was never emptied.

Meanwhile those same six policies differ by 148,857 points over complete games. The benchmark's whole dynamic range is 1.9% of its own mean. There is no room in the metric for the thing it is supposed to measure, and that is an arithmetic fact rather than a noise problem.

How the check works

1. Build a fair way to score a position. Hold the board, the hidden numbers underneath it and the rise clock fixed — that is the position. Hold the visible next disc fixed too, because a player can see it and it is part of the problem the position poses. Then redraw everything else — every future disc, every hidden value of every future risen row — several times, give every policy the same redraws, and score a policy by its average.
2. Prove it is fair, mechanically. If a policy leaked any of the redrawn future, its first move would change on at least one redraw. Across 8 policies × 24 positions × 4 redraws — 768 decisions — none ever did.
3. Fix the honest half/half split before measuring anything. The 128 positions are split into a development half and a sealed half by a rule that is a pure function of the positions' own content hashes, recorded in a manifest with its own hash, so it cannot be quietly reshuffled after a result.
4. Write the pass threshold down first. The check passes if the benchmark's ranking of nine policies agrees with their whole-game ranking at a rank correlation of 0.70 or better.
5. Then run it, and read the verdict off the metric that was named in advance.

What happened

The check failed as preregistered. Rank correlation with whole-game strength is +0.633 across nine policies against a threshold of 0.70 — and across the six real search policies, the ones whose ranking actually matters and which all share a single game cohort, it is −0.257. The benchmark ranks them slightly backwards. No secondary metric rescues it: every alternative measure gives a correlation between −0.64 and −0.03 on those six.

It is a family detector, not a strength meter. Every searching policy beats every non-searching one by a wide margin, and the three weak baselines come out in the right order. Inside the block of real search policies, four of the six comparisons have the wrong sign, and the largest whole-game gap in the whole set — one policy beating another by +133,204 points, a 50% improvement — registers on the benchmark as minus four points.

Its precision makes it more dangerous, not less. The spread it reports between policies is 624 times its own re-measurement noise. An instrument that resolves a quantity that precisely, when that quantity has a negative correlation with the target, produces confident, reproducible, wrong rankings — and reproduces them on demand.

Being cheaper does not help. 128 positions at five redraws each matches the resolving power of a 64-game paired cohort for about 73% of the compute. A 1.4× saving on a measurement that does not predict the target is not a saving.

The split earned its keep on its first use. Applied to the two halves separately, the development half gives a correlation of +0.03 across the six search policies and the sealed half −0.37. A candidate tuned on one half would have lost its margin on the other — exactly the channel the split exists to close, observed immediately.

The failure is repairable, and the repair is not worth building. The mechanism predicts its own fix: lengthen the window until the number of rises can vary. Re-scoring the same positions with the same redraws at a 25-move window moves the correlation across the search policies from −0.600 to +0.800, and on all 128 positions it reaches +1.000 — consistently on both halves of the split and both origins of the positions, where the nine-move version disagreed with itself across all four slices. But a 25-move window is a large fraction of a whole game: a valid benchmark of this kind plays 1.5 to 2.8 times more decisions than the 64-game cohort it would replace, and its cost advantage over whole games ranges from 50× better on one pair of policies to 5,000× worse on another, with nothing observable in advance to say which case you are in. The window that makes it valid is the window that makes it expensive. The recommendation is to keep the machinery for the jobs whole games cannot do at all — a well-posed clairvoyant optimum, controlled board occupancies, a hidden board a teacher can inspect — and retire the ranking claim.

The by-product, which may matter more than the check

With the benchmark's ranking role settled, the same machinery was pointed at a different question: what property of a board actually predicts how many discs it is possible to clear from it? A privileged solver labelled 1,024 positions with the exact clear optimum, averaged over independent guesses of the hidden board and the future, so the label is a function of the public position and is therefore the kind of thing a board-scoring function could learn.

The answer was not the expected one.

The hand-written board scorer is not missing features; it is pointing the wrong way. Reweighting its existing nineteen terms recovers 95% of all the signal any of the 53 candidate properties can supply. Adding new properties on top of a reweighted version buys +0.036. And the specifics are cheap to act on: one term is computed at every node and multiplied by exactly zero despite having a univariate correlation of +0.401 with clear availability; the term with the strongest correlation of all nineteen (+0.627) carries 0.2% of the scorer's direction; and two thirds of the direction is spent on a single term that the fit wants at essentially zero. The only genuinely absent properties that add anything are all one family — how many gray discs a clearing wave can actually reach — which is the same missing quantity the flow ceiling work arrived at from a completely different direction.

This is the origin of the leaf reweight approach. It is worth being precise about what it is and is not: a label that says what is available, not what a legal policy can take, and a demonstration of what predicts a diagnostic — not evidence that any mean score would rise.

What this taught us, and what is still open

• Validate an instrument before citing it. The design document's own gate had never been run, and the numbers already computed on the benchmark turned out to be uninformative about strength — for a stronger reason than "one sample is noisy": the metric does not track the target at that horizon at all.
• Horizon is a first-class parameter of a position benchmark, and this one inherited nine moves from an exact solver's practical limit — a constraint that did not apply to the fair evaluation at all. That single inherited choice is what broke the check.
• Harvest positions from the policy family you intend to rank. The half of the benchmark harvested from a weak policy's games predicts worst.
• For cheap screening, use paired whole games and read the flow rates first. A score's independent unit is the whole game, so 64 of them carry a standard error near 32,000 points; a clear rate is pooled over every move of every game. The finding is explicit that this is a recommendation with a mechanism and not a measured result, and that it costs nothing to test because it only re-analyses records that already exist.
• Still open: whether reweighting the reference board scorer actually raises a mean score. Addendum A says what predicts a diagnostic label; the distance between that and a stronger policy is exactly what the leaf reweight experiment is for.