Drop7 Research
Play
← Concepts

Four ways a program can learn Drop7

N-tuple networks, neural evaluators, policy-gradient reinforcement learning and Monte Carlo tree search — what each one actually does, shown on a real board, and what happened when this repository tried it.

The reference policy here does not learn anything. It looks four moves ahead, averages over the discs it cannot predict, and scores the boards it reaches with a hand-tuned formula — the idea is explained on choice, chance and looking ahead. Over its 64-game reference cohort it averaged about 308,000 points a game, against a target of one million.

So most of the work in this repository has been an attempt to learn something better: either a better opinion about a board, or the column choice itself. Four families were tried. This page explains each one in plain English, shows the mechanism on a real position, and says honestly what happened — because all four have retained results, and none of them displaced the hand-tuned search.

1. N-tuple networks

The simplest learnable evaluator here does not look at the board as a whole. It looks at it through dozens of small windows — four cells in a row, four in a column, a two-by-two block — and asks, for each window, "what have I learned about this exact little pattern?" Each window's four cell values are packed into a short code; the code, together with the rise clock and the next disc, picks one entry out of a large table of learned numbers; and the board's score is simply the sum of those numbers. Learning means nudging the entries that were active when a game went well or badly. There is no matrix multiplication anywhere, which is why an n-tuple evaluator is fast enough to sit at the bottom of a search tree.

a row window333173144 cells across · 28 of thempattern 0788→ one learned numbera column window333173144 cells down · 28 of thempattern 3333→ one learned numbera 2×2 block333173144 neighbouring cells · 36 of thempattern 1091→ one learned number92 windows, one additionEvery window reads four cells andturns them into a four-digit code.The code, the rise phase and thenext disc select one slot in a bigtable of learned numbers. Add the92 numbers up and that sum is theboard's score. Windows that differonly by where they sit share atable, so 92 windows use 17tables.Nothing here is a neural network: it is alookup and a sum, which is why it is fastenough to sit at a search leaf.Board from figure seed 0x5eed031b; the window positions and codes are read off itdirectly. Table layout as implemented in src/core/native/ntuple.hpp.
Three of the 92 windows the native implementation reads on every board. Windows that differ only by position share a table, so the model has far fewer parameters than a table of whole boards would — and it generalises, because a pattern learned in one corner is recognised in another. Layout as implemented in src/core/native/ntuple.hpp.

What happened here. The most complete attempt trained for exactly 50 million transitions and was then opened on its mandatory burned 64-game gate, where the direct n-tuple policy averaged 181,733.422 points and 56.359 moves and the two-boundary rollout built on it fell further, to 113,643.969 points and 37.375 moves — well below the reference search, and a decisive rejection of that family. The full record is on the n-tuple and Q-learning family page.

2. Value and policy neural networks

A neural network can represent things a sum of small patterns cannot: how the whole board's height profile interacts with where the covered discs sit, for instance. Two shapes were tried. A value network takes the public position and returns one number — how promising is this? — which can be dropped in as the search's leaf evaluator. A policy network skips the board score and outputs the choice directly.

A value network: one board in, one number out33317314the public positionis this cell empty?is it a 3?is it covered?is it cracked?next discdrops to riseone input per question per celltwo small hidden layershow promisingone numberThe NNUE ideaA move changes only a few cells, so the first layer is built to be patched — subtract theweights that switched off, add the ones that switched on — instead of recomputing every leaf.
A value network's inputs are yes/no questions about every cell, plus the next disc and the rise clock. NNUE — 'efficiently updatable neural network', an idea borrowed from computer chess — is the observation that a move changes only a handful of cells, so the first layer can be patched instead of recomputed. That matters when the evaluator runs at millions of leaves per decision.

What happened here. Repeatedly, the model predicted well and chose badly. The structured multi-head value network reached a death-within-25 AUC of 0.999 and a lifetime rank correlation of 0.887 on the data it was fitted to, then fell to 0.855 and 0.510 on held-out whole games and failed its preregistered prediction gates before ever playing. A related counterfactual-successor model learned absolute lifetimes well (rank correlation 0.839 on 242 held-out successors) yet picked the right sibling only 15.4% of the time, and as a search leaf lost its screen by 63,196 points a game. The pattern is the same every time and it has its own page: the sibling trap. The family record is at learned values and policies.

3. Policy-gradient reinforcement learning

The third family removes the evaluator entirely. A network reads the position and emits seven numbers, one per column, normalised into probabilities. The program plays with those probabilities, and afterwards the columns that preceded a good outcome have their probability nudged up and the others down. PPO and actor-critic methods are careful versions of that nudge: a critic estimates how good the position already was, so the actor is rewarded only for doing better than expected, and each update is clipped so a lucky batch cannot swing the policy wildly.

A policy: one board in, seven numbers out33317314the public positionInstead of scoring a board,the network scores the sevencolumns directly andnormalises them intoprobabilities. Playing meanssampling or taking thelargest. Training pushesprobability toward the columnsthat led to more score overthe rest of the game.0123456probability of each column · they sum to 1bar heights are illustrative, not a measured policy
A policy's output is a distribution over the seven columns. Nothing about this is specific to Drop7 — which is the appeal, and also the difficulty: the learning signal is one score at the end of a game that may last a hundred moves, and it has to be divided among all of them.

What happened here. These runs trained, and then calibrated below the bar. The primal-dual actor-critic consumed 131,072 training games and, on its mandatory 512-game final calibration, averaged 175,834 points and 55.006 moves; its terminal-risk upper bound was 0.10496 against a fixed 0.02 limit, so the checkpoint was sealed untrusted and no gameplay screen was ever opened. The PyTorch pipeline never reached its policy-gradient step at all: the public behaviour clone meant to warm-start PPO agreed with its depth-2 teacher on only 0.4718 of held-out decisions against a preregistered 0.55 gate, and on the fixed 32-game development cohort it averaged 141,986.938 points and 45.125 moves against fair depth 2's 191,189.344 and 58.688 — so PPO did not run. The experiments live alongside the n-tuple work under n-tuple and reinforcement learning.

4. Monte Carlo tree search

The fourth family changes the search rather than the evaluator. Instead of expanding every branch to a fixed depth, it runs many quick playouts from the current position: pick a promising column, play forward with a cheap policy, see how it went, and record the result back up the tree. Columns with good averages get tried more often, with a bonus that keeps under-explored columns in the mix, so the tree grows lopsided — deep where the game looks interesting, shallow elsewhere. It is the algorithm behind the famous Go and chess programs, and it never needs to enumerate all 49 branches per move that a full-width search does.

after 8 playoutsthe position now111121125% of playouts went to column 4after 32 playoutsthe position now3243125338% of playouts went to column 4after 128 playoutsthe position now651287415858% of playouts went to column 4after 512 playoutsthe position now1293014392352077% of playouts went to column 4Illustrative counts — this figure shows the mechanism, not a measured search. Each playout picksthe child with the best average so far plus a bonus for being under-tried, plays it forward with aquick policy, and reports the result back up the tree.
Each playout descends by picking the child with the best average plus an exploration bonus, plays forward with a quick policy, and reports back. In this repository's implementation the chance events are sampled from a solver-local stream and nodes are keyed by the complete visible position, so no branch can specialise to a future the player has not seen.

What happened here. The honest summary is that the budget was never the problem. The observable-state version failed its held-out gate on top-1 accuracy (11 of 32 roots, 0.34375, against a 0.35 floor) even though it beat the exact depth-3 search on pairwise accuracy and regret on that panel. A much larger version — 65,536 playouts, horizon 64, a fair leaf — was then audited on 12 disjoint held-out roots: its agreement with the depth-4 search's own move rankings rose from 66.67% to 91.67%, while its ability to rank the 25-move outcome got worse. The audit's conclusion is the useful one: "the public D1 rollout and bounded replay reservoir optimize the wrong continuation distribution", so more simulations would buy a better imitation of a short-horizon search rather than a better player. The record is on the tree-search family page.

The technical record

Every number above is ledger-recorded in the experiment history unless noted; the family summaries are in the strategy landscape and the experiment index.

  • N-tuple. Optimistic phase-conditioned n-tuple (approaches/ntuple-rl/optimistic-phase/optimistic-phase-ntuple.cpp), rejected. Training stopped exactly at 50 million transitions across 1,057,844 completed games; the final chunk averaged 176,247 points and 54.811 moves. On the burned 64-game Stage-A cohort 0x3d200000...0x3d20003f, the direct n-tuple policy averaged 181,733.422 points and 56.359 moves and the fixed two-boundary rollout averaged 113,643.969 points and 37.375 moves, against an absolute gate of 300,000 points and 90 moves. No decision timed out and no game was censored, so this is an algorithmic failure, not a resource fallback. Feature layout: 28 row windows, 28 column windows and 36 two-by-two blocks per board (92 active features), sharing 17 tables of 10,000 four-digit pattern codes, keyed also by rise phase and next disc (src/core/native/ntuple.hpp).
  • Value networks. Structured multi-head NNUE (approaches/value-policy-learning/structured-nnue/structured-value-nnue.cpp): 75,395 parameters, 9,800 fitting and 2,132 held-out labels; training MAE 20.391 moves, death-within-25/50 AUC 0.999/0.978, lifetime Spearman 0.887; held-out MAE 29.524 moves, AUC 0.855/0.614, Spearman 0.510. Both the 50-move AUC and ranking gates failed and no screen seed was read. Counterfactual-successor NNUE (same directory): held-out Spearman 0.839 and MAE 3.888 moves on 242 successors, but direct top-action accuracy 15.4%, raised to 30.8% as a depth-3 leaf; on the historical 7,000-point Sequence-scored screen 0x3e870000...03 the NNUE-leaf search lost by 63,196 points and 39 moves.
  • Policy gradient. Primal-dual actor-critic (approaches/ntuple-rl/primal-dual-actor-critic/primal-dual-actor-critic.cpp), rejected at final calibration. 131,072 training games in the sealed 0x3dac0000...0x3dadffff lane, 128 atomic iterations in 339.663 seconds; the mandatory 512-game initial-board calibration averaged 175,834 points and 55.006 moves; four five-move drift upper-95 bounds were 3.1767/1.8588/3.8765/ 1.6098 against a required zero, and the terminal-risk upper bound was 0.10496 against 0.02. Every gameplay, protected and final seed remains unopened by that experiment. PyTorch behaviour-cloning and PPO lab (approaches/ntuple-rl/torch-ppo/train.py), rejected before PPO: 56,484 training and 17,951 held-out states from exact depth-2 play; held-out agreement 0.471840, top-two 0.697788, cross-entropy 1.367013 against a 0.55 agreement gate; on the 32-game development cohort 141,986.938 points and 45.125 moves versus 79,307.875/27.969 for random, 181,846.438/56.281 for fair D1 and 191,189.344/58.688 for fair D2. One permitted correction was also rejected (held-out agreement 0.471784), and the PPO cohorts remain unopened.
  • MCTS. Observable-state stochastic UCT (approaches/tree-search/observable-mcts/observable-mcts-lab.cpp), rejected at its held-out gate: frozen at 16,384 simulations and horizon 32, it scored 0.6498 pairwise accuracy and 28,420 mean regret on 32 disjoint roots versus exact depth 3's 0.6418 and 44,142, but top-1 was 11/32 = 0.34375 against a 0.35 gate. Scaled audit (approaches/tree-search/observable-mcts/observable-mcts-scaled-audit.cpp), rejected before gameplay: 65,536 simulations, horizon 64, 16-outcome reservoir, one public fair leaf. On 12 corrected-score held-out roots it reached 25.00% top-1 and 56.78% pairwise against the old configuration's 25.00% and 61.02%, with normalized regret rising from 0.29450 to 0.37251; against corrected fair-D4 root Q on the same states, top-1 rose from 66.67% to 91.67% and pairwise from 66.10% to 74.58%. Twelve roots is a very small panel, and the audit opened no gameplay seed.

The common thread

Three of these four families are ways of learning a value, and all three ran into the same wall. The status summary states it as a durable conclusion: "State-value accuracy is not action-ranking accuracy. Training data must cover legal siblings or use an objective designed for relative action value." A model trained on the moves a policy actually played is confident about boards that policy liked and guessing about the six it skipped — which is precisely the comparison it is asked to make at play time. That failure has its own page: evaluating a board, and the sibling trap.

The fourth family, tree search, sidesteps the training-data problem and hits a different one: a search is only as good as the policy it uses to continue past its own horizon, and a cheap continuation makes a confident but wrongly-aimed estimate.

None of this says these methods cannot work on Drop7. It says the tested configurations did not, for reasons that were diagnosed rather than guessed — and the diagnoses point at the shape of the training data and the quality of the continuation, not at model size or search budget. That argument is followed through on is more computation the answer?.

← Cheating on purpose: oracles, teachers and students