The search plateau

Yesterday ended with three bets running overnight on a saturated machine. All three were, in different ways, the same bet: if we spend more compute, does the planner get better?

The answer, consistently, is no — with one exception that turns out to explain the rest.

1. 17:43Depth-3 reveal-sampling ladder begins.
2. 23:32Depth-4 reveal-sampling continuation arm starts.
3. 01:38Leaf-reweight arms report a monotone negative across six configurations.
4. 05:43Depth-5 with five chance strata completes 64 games; the contrast lands far below the noise floor.
5. 09:45GPU: fully-trained afterstate model fails its override gate.
6. 10:45GPU: the same model family cannot reproduce the planner's own ordering even from exact labels.
7. 11:07Playable web console with a parity-tested browser solver lands from a parallel session.
8. 11:24A stronger-teacher retrain is already running on the GPU, and the depth-5 cohort is 24 games in.

The whole factorial, in one table

Every cell below plays the same seeds under the same scoring, differing only in search depth and in whether the model of the unknown next disc is exact (7 chance strata) or approximate (5). All are complete 64-game cohorts except the depth-5 seven-stratum cell, which was stopped by decision at 32 games.

Search depth	5 strata (approximate)	7 strata (exact)
2 plies	249,641	265,294
3 plies	305,051	312,327
4 plies	297,327	398,498
5 plies	288,704	411,874 (32 games)

Read the two columns separately and they tell opposite stories.

With an approximate chance model, the best depth is three. Adding a fourth ply loses 7,723 points; adding a fifth loses more. Deeper search makes the policy worse, because it is optimising ever more carefully against a distribution that is wrong.

With an exact chance model, the fourth ply is worth +86,172 points, with a 95% lower bound of +26,605 — statistically significant, winning 40 of 64 paired games. It is the single largest verified improvement in this repository.

Chance-exactness does not merely add points to the search. It flips the sign of the depth gradient.

The fifth ply is a different story, and not the one I first reported: it turns out to be unmeasurable rather than worthless. That is the next section.

Depth 5: not answered, and not answerable this way

This is the section I have had to rewrite. At 16 games the fifth ply measured −1,581 and I reported the depth axis as closed. The cohort was then stopped at a 32-game chunk boundary, and the extra 16 games moved the estimate to +23,367 — the sign flipped.

Paired contrast	n	Δ score	Paired sd	Detection floor
d4 − d3, exact chance	64	+86,172	298,877	61,457	measurable
d4 − d4, 7 vs 5 strata	64	+101,171	268,413	55,192	measurable
d5 − d5, 7 vs 5 strata	32	+123,613	—	95,207	measurable
d5 − d3, exact chance	32	+86,397	—	97,211	below the floor
d5 − d4, approximate chance	64	−8,624	228,827	47,052	below the floor
d5 − d4, exact chance	32	+23,367	371,351	107,988	below the floor

The property holds across all six contrasts in the factorial, with no exceptions in either direction: every result that reaches statistical significance sits above its own detection floor, and every one that does not sits below it. Note the third row especially — d5 − d3 is a large effect, +86,397, and still fails, because at n=32 the floor for that contrast is 97,211. Size alone does not rescue you from a small sample.

The null results are not evidence of no effect. They are non-measurements. Paired game-to-game variance in this game is enormous — the five largest single-seed swings in the depth-5 comparison are −1,002,862, +958,985, −678,455, +592,546 and −577,069. Individual games differ by more than twice the entire cohort mean. Against that noise, a 64-game paired cohort can only resolve effects larger than roughly 50,000 to 60,000 points. Anything smaller is invisible by construction.

open

What the fifth ply would cost to measure

684 games

To give the observed +23,367 a one-sided 95% bound would take 684 paired games at this variance. At the arm's measured throughput — 1,647 seconds per game on 14 threads — that is about 13 wall-days, and realistically 8 to 18, since the two chunks differed by 2.3x depending on what else was running.

Finishing the cohort to its planned 64 games would not have helped: the standard error would still have been about 46,000, twice the point estimate. This contrast cannot be resolved by any cohort this project can afford.

So the correct statement about depth 5 is not "it buys nothing". It is: its effect, in either direction, is smaller than this experimental design can see. What can still be said is economic rather than statistical — even the optimistic point estimate is +23,367 for 35.6x the work per move, which is a bad purchase whatever its true sign.

positive

Exact chance modelling replicates at depth 5

+123,613

One result got stronger, and it is the one that matters. Holding depth fixed at five and moving from an approximate to an exact chance model is worth +123,613 points, 95% lower bound +32,575, W–T–L 19–0–13 — statistically significant, and a replication at a new depth of the same effect measured at depth 4 (+101,171).

Every significant result in this entire factorial is a chance-resolution contrast, or the fourth ply given exact chance. Not one is a depth contrast on its own.

The plateau

The same picture appears again if you ignore the labels entirely and just plot score against compute. These arms all use the exact chance model and differ only in how they spend the budget — d is depth, M is how finely the hidden gray-disc reveals are sampled.

Arm	Mean score	Work / move
d3 · M=1	312,327	156,834	cheapest
d3 · M=3	337,306	1,045,719
d3 · M=6	376,442	4,244,020
d4 · M=1	398,498	4,956,614	reference
d3 · M=12	349,345	13,506,434
d4 · M=2	356,548	20,178,327
d5 · M=1	411,874	176,536,117	32 games

The cheapest and most expensive arms differ by a factor of 1,085 in compute and about 32% in score — and the ordering is not monotone: two arms costing 3x and 4x the reference score below it. Every arm above roughly five million units of work per move sits within the reference's noise floor, in both directions. That is the shape of a plateau, not of a curve still climbing.

Two arms make the point sharply. d3 · M=6 and d4 · M=1 cost almost the same (4.24M versus 4.96M) and score almost the same (376,442 versus 398,498), despite spending that budget on completely different things — one on modelling hidden information, the other on looking further ahead. They land in the same place because they are the same place.

Modelling the hidden discs more finely: it saturates

Drop7 hides the value of gray discs until they are revealed. M is how many distinct reveal outcomes the planner samples at each chance node — M=1 is a single guess, and at depth 3 twelve samples is full joint coverage.

Arm	Joint coverage	Mean score	Δ vs M=1	95% lower	W–T–L
M=1	8.3%	312,327	—	—	—
M=3	42.9%	337,306	+24,980	−23,451	32–0–32
M=6	85.7%	376,442	+64,116	+7,475	36–0–28
M=12	100%	349,345	+37,019	−25,076	30–0–34

positive

Reveal sampling pays at depth 3, and stops paying before full coverage

+64,116 at M=6

M=6 is worth +64,116 points with a 95% lower bound of +7,475 — one of the few statistically significant positives this project has. Going all the way to complete joint coverage at M=12 does not extend the gain: measured directly against M=6 it is −27,097 (95% bounds −83,807 to +31,209, W–T–L 28–0–36).

Because that step sits inside noise, this is saturation, not regression — the honest claim is that the reveal axis is exhausted by about M=6, not that full coverage is actively worse. But the turnover is visible in four separate quantities at once: score, moves, clears per move and reveals per move all peak at M=6. Only mean occupancy keeps improving all the way to full coverage.

The obvious next question is whether the same refinement stacks on top of the fourth ply. The depth-4 arm scores −41,950 against the depth-4 control, and here I have to be careful about what that does and does not say.

dead end · recorded result

Stacking reveal sampling on top of the fourth ply

cost: 4.07x the work for 64 games

−41,950, 95% lower bound −100,137, W–T–L 28–0–36 — for 4.07x the work per move.

Two honest caveats, both of which I initially got wrong. First, this is not significantly negative: the one-sided 95% upper bound is +17,541, so the correct reading is "buys nothing measurable", not "harmful". Second, it is not the same dose. The depth-4 arm tested M=2 — 28.6% joint coverage — because the wider arms were never affordable at depth 4. At depth 3 the comparable dose M=3 was itself not significant. So this rejects doubling the reveal samples on top of ply 4; it does not test the dose that actually worked at depth 3.

What survives is the weaker and better-supported claim: depth and chance resolution substitute for each other rather than compounding. The clean evidence for it is the equal-work pair, where the doses are irrelevant because the budgets match — d3 · M=6 at 4.24M work per move scores 376,442, and d4 · M=1 at 4.96M scores 398,498. Two entirely different ways of spending the same compute, landing in the same place.

The leaf evaluation is already well designed

Earlier I characterised the search's leaf-evaluation weights as "pointing the wrong way" — they correlate only weakly with how many clears a position can actually achieve. That reading was wrong, and the experiment I commissioned on the strength of it returned a clean, monotone negative.

dead end · recorded result

Refitting the leaf toward achievable clear rate

cost: six arms, 64 paired games each

Six weight vectors, ordered by how well each predicts achievable clears. Score falls monotonically as the fit improves. The fully-fitted vector scores 161,316 — a loss of −237,182 points, winning 7 of 64 games.

Clears per move drop from 2.0571 to 1.6019, i.e. a leaf that better predicts available clears produces a policy that achieves 22% fewer of them.

The surprise was in the tails. The risk I flagged — that this would damage death-avoidance — did not materialise; every arm improved the worst games. What died was the ceiling. Maximum score fell from 1,341,287 to 342,164 and the standard deviation from 254,414 to 54,703. The refitted policy is more consistent and cannot win big.

The correct interpretation inverts my original one: the leaf's job is to price what four plies of search cannot see. The search already computes short-horizon achievable clears directly and exactly. A leaf that duplicates that work adds nothing and displaces the information that was actually load- bearing. Weak correlation with achievable clears is evidence of good design, not a defect.

The GPU line reached the same wall from the other side

A second contributor spent the night training on the machine's integrated GPU, pursuing the opposite hypothesis: that a learned evaluator could see structure several row-rises out that no tractable search can reach.

dead end · recorded result

Training the afterstate model to completion

cost: ~5.5 GPU-hours, 44M row-updates

The previous checkpoint might simply have been undertrained. It was trained to 22 epochs — twice the previous budget, cosine schedule completed, training loss duly improving from 0.630 to 0.584.

On the same held-out positions it ranked worse: top-1 accuracy 0.361 versus the earlier checkpoint's 0.424, normalized regret 0.281 versus 0.241. Training loss down, held-out ranking down — textbook overfitting. The frozen gate then failed on fresh data, and failed informatively: the override helped in one half-fold (+0.0228) and was actively harmful in the other (−0.0297). Direct override by this model family is now closed.

dead end · recorded result

Learning the planner's ordering from exact labels

cost: 291,890 labelled afterstates over 8,639 roots

This is the sharper result. Rather than asking the network to be better than the search, ask it only to reproduce the search — with every advantage: successor-closed coverage, every legal sibling labelled, exact search values, and label completeness of 1.0.

It reached top-1 agreement of 0.375 against a required 0.60. For scale, the cheapest possible exact search — a single ply, depth 1 — agrees with depth 4 at 0.486. The trained network is a worse approximation of the planner than the planner's own shallowest version.

That reframes what has been blocking every learned policy in this repository. The standing explanation was insufficient data coverage: models saw the action that was played, not its siblings, so they could not rank alternatives. This experiment gave a model perfect coverage and exact labels, and it still failed. The obstacle is the representational capacity of a compact board evaluator for a 4-ply search-value function — not the data.

What actually shipped

positive

A playable console with a browser-native solver

A parallel console session landed an interactive Drop7 board and a /play route where the search runs in a web worker — with parity tests against the repository engine, so the browser plays the same game the research does.

Where this leaves us

Two independent lines of attack converged from opposite directions: more search does not measurably help, and a learned evaluator cannot match the search we already have. The fair-planner family sits at roughly 400,000 points against a target of 1,000,000.

The word measurably is load-bearing, and today taught us how much. A 64-game paired cohort in this game cannot see an effect below about 50,000 points. Every null result in this factorial is smaller than that. We have not shown that deeper search is worthless; we have shown that if it is worth anything, it is worth less than we can detect for a cost we can pay — and that chasing sub-50k effects with whole-game cohorts is not a viable way to spend this machine.

Why the student failed has exactly three explanations left, and they are separable: the teacher was too weak, the student was too small, or the four-ply search-value function genuinely cannot be represented by a board evaluator of any practical size. The first two are cheap to test and are being tested. The third is the interesting one, because if it is true then the way forward is not a better approximation of this search — it is a different objective.

dead end · recorded result

Retraining the student against a stronger teacher

cost: ~2.1 GPU-hours

Ran and failed during the day. Last night's student learned from a weak depth-1 continuation, which its own record named as the limitation, so this run relabelled the corpus with a fair depth-2 teacher. The gate was fixed before the data was read: top-1 agreement of at least 0.4616 in each half-fold, closing half the gap to fair D4.

It reached 0.3365 — below the previous student's 0.4245 and level with exact D1's own 0.3339 on the same panel. The override gate failed too, helping in one half-fold (+0.0110) and hurting in the other (−0.0089). What did pass is worth keeping: label stability rose to 0.806 where the earlier run at the same scenario count scored 0.446 and was ruled inconclusive. The stronger teacher fixed the labels; it did not fix the ranker.

Its author disclosed the confound that matters: the model was trained on depth-2 continuation outcomes but graded against depth-1 continuation outcomes, frozen that way for comparability. Part of the regression may be that mismatch rather than teacher quality, and the corpus was also cut short at 6,535 of 8,192 roots by a wall-clock stop.

proposed

Refit the leaf against remaining lifetime

The refit that failed targeted achievable clears. The quantity that actually correlates with score at r = 0.9995 is remaining lifetime. That is the label the leaf should be fitted against, and it has not been tried.

proposed

Test whether model capacity is really the binding constraint

The failed student was a 3.4M-parameter network on an integrated GPU. The claim "a compact evaluator cannot represent this function" is only established for compact evaluators. It predicts that a materially larger model does better, and that prediction is cheap to falsify.

proposed

Stop measuring small effects with whole-game cohorts

Not "stop buying depth" — that was the conclusion I reached this morning and it claimed more than the data supports. The defensible version is narrower: any change worth less than about 50,000 points is invisible to a 64-game paired cohort, and depth 5 costs 35x the work to produce an effect we could not measure. Either find effects large enough to see, or find a lower-variance measurement than complete games. Both are better uses of the machine than another point on the depth axis.

Making the public competition reproducible

positive

Competition lifecycle and AI seeding are now scripted

Codex added a checked-in catalog for the current and archived public games, a preview-first command for starting, activating, archiving, and listing games, and an idempotent DynamoDB seeder for the registered public-information AI contenders. Every AI record carries its packed moves, exact replay checksum, information-boundary flag, and a persisted HTTPS link to the approach page that explains it. Archived leaderboards remain queryable by their immutable game key.

The D4 entry here is explicitly the existing TypeScript playground analogue, not a source-identical port of the frozen native fair-D4 research comparator. These single scripted-game scores remain demonstration data, never research evidence.

closed

Deploy the policy-aware reader, then seed production

owner: repository owner

The policy-aware reader was deployed from clean commit c215cf4, then the production seeder conditionally stored all nine registered public-information AI contenders. A DynamoDB query confirmed that every policy row points back to that clean revision and carries an HTTPS research link, and a live fetch showed the AI names and approach links on the public leaderboard. A second Greedy-only invocation returned already-present, confirming that the production workflow does not overwrite its immutable seeded record.

These are still single scripted-game playground entries. Deployment changes their visibility, not their scientific evidence tier.

Evening: evolving the leaf at the depth it is used

A new line of work opened in the evening, on the dev branch, from a different model (claude-fable-5 in Claude Code) with Kimi K3 as its adversarial reviewer. The question it takes up is the one this morning's leaf finding left standing: the nineteen leaf weights were hand-set for a depth-3 search under the old scoring and have never been refitted at depth 4. Two earlier tuning attempts failed under conditions that each explain failure on their own — a depth-3 fit transferred to depth 4 unchanged, and fixed tuning games that a heavy-tailed optimiser can memorise. Both channels can be removed at once: evaluate fitness with complete depth-4 games, and give every generation a fresh block of seeds.

running

CMA-ES over the eighteen active leaf weights, depth-4 fitness, fresh seeds per generation

owner: claude-fable-5 (Claude Code)

Theory TH-20260822-depth-native-leaf-weights-caa6f8ba, experiment EX-20260822-leaf-cmaes-d4s7-4f5f462a, frozen before the first training seed was read. The fast engine's leaf now takes its weights as data (approaches/lifetime-objective/leaf-evolution/), with the search generated from fast-search.hpp by a checked substitution. Six CHECK gates passed with zero mismatches, and a pilot reproduced the recorded reference cohorts seed for seed while measuring the cost of a decision on this machine: about 0.29 CPU-seconds at depth 4 with five chance samples, 1.03 with seven. The run (RUN-20260822T014412Z-a0c63063) evolves at the five-sample configuration — the contract comparator, and 3.7× cheaper per game, which buys roughly forty generations overnight instead of twelve — with a preregistered seven-sample transfer arm on the same held-out seeds as a secondary diagnostic. The candidate is the CMA distribution mean at the last generation, never the best sampled individual, and it meets one 64-game held-out screen (0xa52b0000+) exactly once.

Fresh blocks per generation correctly prevent seed overfitting — but they convert overfitting into drift, not into signal.

Kimi's pre-run review (runs/RUN-20260822T014331Z-1803c746/kimi-k3-review.md) found no information-boundary or seed-role violation, checked every CMA-ES constant against Hansen's tutorial, and verified the pilot's reproduction claim. It then put the odds of the gate passing at about one in ten, for a reason worth recording before the data arrive: finding-14's perturbation arms at depth 4 had a paired standard deviation near 360,000 points, so a 32-game fitness estimate is far noisier than the differences CMA-ES needs to rank, and the only direct measurement of the local surface at deployment depth was flat to negative. It also caught three process defects, all fixed before the first seed was opened: the new leases were missing from the lease map's table, the held-out screen command was not scripted (screen.sh now is), and the "Student-t" bound in compare.py was using the normal quantile. Two of its alternatives are recorded as follow-ups — a two-stage allocation inside each generation, and perturbing in units of feature standard deviation rather than of the frozen weight — and one is declined: ranking candidates by paired wins would change the estimand away from the mean score the theory is about.

proposed

An orchestrator for this loop

owner: claude-fable-5 (Claude Code)

./do-research (in scripts/do-research/) turns the contract into a loop a machine can run unattended: a planner model proposes one preregisterable experiment from the repository's own records, a second model family critiques it, the orchestrator registers theory, experiment and seed leases through the allocator — which can never touch a protected or final range — a headless claude -p implements and runs it inside hard caps, opencode audits the result, and the planner decides whether to continue. Untested on a full iteration as of this entry; the first end-to-end run is owed.