Offline gate: calibrated top-two near-tie override of fair D4 by the frozen afterstate model
The third pilot established that the learned board-scorer cannot replace the four-move search as the thing that picks the move. This experiment asks a much smaller question: could it still be useful in the one situation where the search is least sure of itself?
Often fair D4 rates its top two columns almost identically. Those near-ties are where a four-move estimate is most at the mercy of which futures it happened to sample. The candidate here is unchanged fair D4 with one conservative addition: at a near-tie — the top two within 500 points, a threshold fixed from already-read data before this experiment existed — consult the frozen third-iteration network, and swap to the search's second choice only if the network's advantage survives a resampling test. The comparator is unchanged fair D4. Nothing else about either player differs, and again no games were played: both were graded on how well they ordered columns at fresh held-out positions.
"Pass" required the override to cut average regret by at least 0.01 in each half of the held-out set, to fire on at least 5% of eligible positions, and to reproduce byte-identically across two runs.
What happened. A valid run with a fail outcome — but a narrow one, and
the record says so explicitly. The rule fired often (38.3% of positions were
near-tied; the override triggered on 37.0% of those), it was deterministic, and
it improved regret in both halves: by 0.0351 in one and 0.0075 in the other.
The frozen bar was 0.01 in each, so the second half missed by a quarter of a
percentage point and the gate fails. The record's own reading is that the
mechanism works as designed and that this is the first preregistered held-out
test here where a learned model's intervention improved on fair D4 at all, with
the checkpoint's ranking quality as the bottleneck. It also warns, in the
limitations, that the miss "should be read as a narrow failure, not as evidence
of no effect."
The run was valid; the outcome was fail, at the a pilot — a small run to find bugs and project cost, not a strength claim level. Of 7 preregistered checks, 6 passed and 1 failed.
The calibrated top-two near-tie override of fair D4 by the frozen iteration-3 afterstate model FAILED its frozen gate by a narrow margin, with genuinely positive signal. On 2,689 fresh held-out roots, 1,030 (38.3%) were near-tied at the frozen 500-point threshold and the override fired on 37.0% of those. On eligible roots the override reduced mean normalized regret versus unchanged D4 in both half-folds (half1 0.2484 vs 0.2835, +0.0351; half2 0.2343 vs 0.2418, +0.0075; pooled +0.0214), but the frozen criterion required at least +0.01 in EACH half-fold, and half2 fell short. On the 84 decisive eligible roots the improvement was larger (+0.0426: 0.0880 vs 0.1306). Label stability passed (decisive Spearman 0.814), model quantile calibration passed (0.861), the gate is deterministic (byte-identical repeated reports), and the corpus is successor-closed. Interpretation: the conservative-override MECHANISM works as designed (fires often, calibrated, positive in both halves) and this is the first preregistered held-out test in this repository where a learned model's intervention improved on fair D4 at all; the CHECKPOINT's ranking quality is the bottleneck, consistent with the model being undertrained (11 of 20 epochs) and trained on a weak D1-continuation teacher. Per the frozen failure action, no further override variants of this same checkpoint are opened; a retry requires new evidence such as a fully trained or stronger-teacher model under the same frozen rule.
Limitations recorded with the result
- The evaluation target is the H40 phase-greedy-D1 continuation outcome - the same label family the model was trained to predict; D4 is compared against the same target, but the offline gate cannot measure real gameplay interaction, which only a SCREEN tier can.
- The model was undertrained (11 of 20 epochs) and used a weak D1-continuation teacher; both are documented as the likely bottleneck and motivate any retry.
- The near-tie threshold (500) and regret margin (0.01) were frozen choices; the half2 miss (0.0075) is close to the margin and the result should be read as a narrow failure, not as evidence of no effect.
- Single machine profile; FP32 on the shared-memory iGPU.
What it closes: no further override variants of this checkpoint without new evidence. What it leaves open: the same frozen rule applied to a fully trained model, or one taught by a stronger continuation player. And what it cannot speak to at all is real play — an offline ranking test cannot measure how these choices interact over a whole game, which only a paired gameplay tier could.
The registered protocol
At roots where fair D4's top two actions are near-tied (Q gap <= 500 points, the bottom ~37% of the top-two gap distribution measured on already-read development labels), the frozen iteration-3 distributional afterstate model can identify the better action more reliably than D4's own ordering, because near-ties are where D4's 4-ply estimates are most noise-sensitive while the model's calibrated H40 outcome distributions carry independent long-horizon information. The override policy (D4, plus swapping to the second action only when the model's paired 256-scenario advantage has a positive 95% bootstrap lower bound) will show lower mean normalized regret than unchanged D4 on eligible fresh held-out roots.
approaches/afterstate-learning/distributional-afterstate/override-gate.pyapproaches/fair-expectimax/reference/fair-only-depth4.cppPrimary metric
Mean normalized regret of the override policy versus unchanged fair D4 on override-eligible fresh held-out roots, against the scenario-mean H40 outcome ranking (K=256, phase-greedy D1 continuation), reported per origin-hash half-fold.
Statistical unit: whole-origin
Pass criteria, fixed in advance
- Corpus is successor-closed: every non-trivial harvested root has labels for every legal sibling under all 256 scenarios.
- Label stability floor: decisive-root scenario-half Spearman >= 0.5; below this the outcome is inconclusive.
- On eligible roots, override-policy mean normalized regret <= D4 mean normalized regret - 0.01 in EACH half-fold.
- Override rate >= 5% of eligible roots (a rule that never fires proves nothing).
- Nominal 76.5% quantile interval coverage of the frozen model on fresh afterstates within [0.66, 0.87] (sanity that the model is in-support).
- The gate script is deterministic: two runs produce byte-identical reports.
- All existing CHECK-tier self-tests pass before any label is inspected.
On pass: Preregister a SCREEN gameplay experiment: 32 paired public-development games, native fair D4 with the frozen override rule (model inference over stratified afterstate samples at near-tied roots only) versus unchanged fair D4, with per-game result rows and the standard cohort summary.
On fail: Record a valid negative (or inconclusive) result. A failure here closes the direct-override use of this model: no further override variants of the same checkpoint without new evidence.
Data and reuse
The model is the frozen iteration-3 checkpoint (trained only on seeds 0x5da70000-0x5da700ff, training-role). The near-tie threshold 500 was fixed from already-read iteration-2/3 comparator labels before this experiment's data existed. Evaluation roots come from fresh seeds 0x5da70300-0x5da7033f (64 origins), read for the first time by this experiment's single gate evaluation. No protected or final data is read.
seed leases: SL-20260820T083000Z-5da70000
What happened
The calibrated top-two near-tie override of fair D4 by the frozen iteration-3 afterstate model FAILED its frozen gate by a narrow margin, with genuinely positive signal. On 2,689 fresh held-out roots, 1,030 (38.3%) were near-tied at the frozen 500-point threshold and the override fired on 37.0% of those. On eligible roots the override reduced mean normalized regret versus unchanged D4 in both half-folds (half1 0.2484 vs 0.2835, +0.0351; half2 0.2343 vs 0.2418, +0.0075; pooled +0.0214), but the frozen criterion required at least +0.01 in EACH half-fold, and half2 fell short. On the 84 decisive eligible roots the improvement was larger (+0.0426: 0.0880 vs 0.1306). Label stability passed (decisive Spearman 0.814), model quantile calibration passed (0.861), the gate is deterministic (byte-identical repeated reports), and the corpus is successor-closed. Interpretation: the conservative-override MECHANISM works as designed (fires often, calibrated, positive in both halves) and this is the first preregistered held-out test in this repository where a learned model's intervention improved on fair D4 at all; the CHECKPOINT's ranking quality is the bottleneck, consistent with the model being undertrained (11 of 20 epochs) and trained on a weak D1-continuation teacher. Per the frozen failure action, no further override variants of this same checkpoint are opened; a retry requires new evidence such as a fully trained or stronger-teacher model under the same frozen rule.
- ✓Successor-closed corpus (all legal siblings x 256 scenarios) — observed: 4,858,880 rows over 2,793 roots; completeness 1.0
- ✓Label stability floor: decisive-root scenario-half Spearman >= 0.5 — observed: 0.8144
- ✕Eligible-root override regret <= D4 regret - 0.01 in EACH half-fold — observed: half1 +0.0351 (passes); half2 +0.0075 (below the frozen 0.01 margin); pooled +0.0214
- ✓Override rate >= 5% of eligible roots — observed: 37.0%
- ✓Quantile interval coverage within [0.66, 0.87] — observed: 0.8606
- ✓Gate script deterministic (byte-identical repeated reports) — observed: two runs byte-identical after moving wall time out of the report
- ✓All existing CHECK-tier self-tests pass before any label is inspected — observed: SELFTEST OK (9 checks)
Recorded metrics
- The evaluation target is the H40 phase-greedy-D1 continuation outcome - the same label family the model was trained to predict; D4 is compared against the same target, but the offline gate cannot measure real gameplay interaction, which only a SCREEN tier can.
- The model was undertrained (11 of 20 epochs) and used a weak D1-continuation teacher; both are documented as the likely bottleneck and motivate any retry.
- The near-tie threshold (500) and regret margin (0.01) were frozen choices; the half2 miss (0.0075) is close to the margin and the result should be read as a narrow failure, not as evidence of no effect.
- Single machine profile; FP32 on the shared-memory iGPU.