The scripted-round benchmark
The leaderboard answers one question fairly: given identical luck, which policy scores more? Ordinary seeded play cannot promise that — a policy that opens more gray discs would consume more random numbers and quietly receive a different disc sequence. The scripted-round format removes that problem by deciding every random outcome before the game starts.
Two predetermined tapes
Each round (the well-known Gauntlet 01–08 suite ships with the repository) is a small JSON document with two tapes:
discs[m] is the visible disc at move m. Indexed by move
number, so every policy sees the same sequence no matter how it plays.
latentRows[r][c] is the hidden value that the covered disc in
column c of rise r will reveal — fixed before the game, carried with the
disc through gravity and rises.
Because rises always happen after moves 5, 10, 15, … for every policy, the r-th latent row is consumed by the r-th rise no matter what the policies do. A gray disc's value is therefore a property of the disc, not of the moment it happens to be revealed.
Every covered cell in the opening row already has its value written down in latentRows[0]. Revealing it can only ever produce that value.
What a run records
Each policy × round pairing produces a complete game record: final score, moves survived, censor flag at the 2,000-move cap, numbered clears and covered reveals per move, maximum chain depth, illegal choices (a legal fallback is played and the lapse is counted), wall time, and a SHA-256 trajectory checksum. The full move-by-move frames power the replay viewer — click any score on the leaderboard to watch the game.
Honest limits
Policies are also compared at fixed algorithmic work (bounded search budgets), not fixed wall time, so a faster machine does not make a policy look stronger. Policies that read state outside the strict public interface (level or move number) are flagged with an "extended state" badge.
Reproduce it
npm run bench # default policies, all 8 rounds
npm run bench -- --policies greedy,expectimax-d2 --rounds gauntlet-03
npm run bench -- --all # adds slow reference D3/D4
The output lands in web/data/leaderboard.json with one replay file per game
under web/data/replays/. Re-running the same configuration reproduces every
score and checksum exactly.