1. Ask what makes a strong player
Connect Four has only seven possible columns, but a good move can depend on a long sequence of replies. Learning to take an immediate win is different from recognizing a position that eventually forces one. Our experiment studies how a policy/value network learns useful move preferences, and how much extra strength comes from planning around those predictions.
We distinguish three players: the raw neural policy, the network with Monte Carlo tree search, and the network with a native proof/search controller. All use the same selected network, but they do different work before choosing a move. A score achieved by the complete controller cannot be advertised as the strength of the neural file alone.
2. Represent the board from the player’s perspective
The selected network receives 98 visible, board-derived values: 42 cells for our pieces, 42 for the opponent’s pieces, seven hypothetical immediate-winning columns for us and seven for the opponent. Cells are ordered row by row, top-left first. “Us” always means the player whose turn it is, so the same board pattern can be learned for either piece color.
The two piece grids and two tactical-column grids, repeated over board rows, form four spatial planes. A separate legal mask excludes full columns. Engineered winning/threat features supply tactical knowledge explicitly; the network is not learning every game rule from scratch. It never receives hidden randomness or the opponent’s identity.
One shared representation, two heads
A 3 × 3 convolution maps four channels to 32, followed by three residual blocks, each containing two 32-channel 3 × 3 convolutions. A residual block adds a scaled transformation back to its input, letting the network refine features rather than replace them completely. The residual scale here is 0.1. There is no BatchNorm, dropout or recurrence.
Policy and value heads each project 32 channels to two with a 1 × 1 convolution, flatten the spatial features, and use a 64-unit ReLU layer before the output. The complete network has 69,102 learned parameters. Two trainable tactical coefficients also influence policy logits; these are learned scores rather than a separate rule that overrides the selected action.
- Policy · seven logits
- One score per column. Raw inference applies the legal mask and takes the highest-scoring available column. Policy predicts which action to prefer.
- Value · one number in [−1, +1]
- A tanh output estimates the outcome from the current player’s perspective. Proved wins, draws and losses correspond to +1, 0 and −1. An intermediate prediction is an estimate, not an exact proof or a calibrated win probability.
3. Turn stronger search into teaching data
- Collect positionsGames and difficult roots
- Ask teachersPolicy targets & proofs
- Train both headsMask unknown outcomes
- Confirm freshSame starts & budget
Search distillation trains a fast neural model to imitate a more expensive teacher. Games provide reachable positions; stronger search supplies action targets and, when available, proved outcomes. Self-play provides experience, but this selected training phase is supervised learning from search targets, rather than a new PPO update.
Build a spatial parent, then continue it
The spatial parent started fresh with seed 173. A mixed collection of 1,024 complete games produced 32,934 mirrored examples from 16,467 canonical positions; 21,932 examples carried proved values. The selected parent run trained for 300 total epochs with masked policy cross-entropy and proved-value mean squared error, using AdamW at learning rate 0.0005, batch 512, weight decay 0.0001 and gradient clipping at 2.
The accepted continuation started from that parent with training seed 193. It combined repaired historical replay, 4,096 fresh games and certified tactical examples. Of those fresh games, 1,647 restarted from archived difficult training roots. That curriculum concentrates useful experience on situations the earlier player struggled with, instead of repeatedly collecting only easy openings.
Teachers included bounded depth-10 search and an exact teacher, alongside the parent policy, tactical opponents and custom search. The final dataset contained 191,676 rows from 94,941 canonical positions, with 161,662 proved value targets. Horizontal reflections provide corresponding mirrored boards and action labels; they are also excluded when checking for evaluation overlap.
Use reliable labels and mask the rest
Immediate wins and genuinely unique safe blocks take precedence in policy targets, including positions with a proved eventual loss. This repaired 1,682 historical policy rows. “Safe block” means an action that prevents an immediate loss when such a unique action exists; it does not necessarily guarantee a winning future.
Training uses legal-masked cross-entropy for policy, mean squared error for proved values, and a tactical logit margin of 2. Unknown outcomes have zero value-loss weight. An interrupted search is not a draw; turning an unknown outcome into a zero label would teach false certainty. A position can still contribute a policy target without contributing a value target.
Each of 60 epochs presents every replay row plus extra tactical replay equal to half the dataset. AdamW uses learning rate 0.00015 for ordinary parameters and 0.003 for the tactical coefficients, weight decay 0.0001, batch 512 and gradient clipping at 2. Training ran locally on Apple MPS.
An epoch is one pass through the specified records; a batch groups records for a gradient update. Learning rate controls the update scale. AdamW adapts updates per parameter and applies weight decay, which discourages unnecessarily large weights. Clipping limits unusually large gradient steps. These choices control fitting behavior; fresh game comparisons still decide whether the fitted model is useful.
4. Use prediction to guide planning
Raw policy chooses once from the seven logits. Monte Carlo tree search (MCTS) explores continuations before deciding. Its PUCT rule combines estimated value with an exploration bonus informed by policy priors and visit counts. A promising, little-explored action can receive more attention; repeated visits then refine the comparison.
Our neural search uses 256 simulations, exploration coefficient 1.5, prior temperature 1 and a 5% uniform support floor. The floor leaves some prior support for every legal move, so a very low neural score does not eliminate a move from consideration. Simulation count is a work budget, not a proof that the best move has been found.
- Bounded exact querySeek proved action outcomes
- Terminal searchLook for forced wins
- Neural PUCTPlan when proof is incomplete
The selected full controller first uses exact search capped at one million nodes, with the network helping order actions. Partial proofs remove proved losing root actions while retaining unresolved ones. When no complete proof is available, it tries depth-12 terminal-win search with a 150,000-node cap, then neural PUCT. Proof transpositions reset per game; the controller observes only public move history.
In one audit, exact proofs handled 5,920 of 6,361 decisions, about 93.1%. That makes search a major contributor to the full player’s strength. A neural prediction is approximate; an exact proof establishes an outcome under the game rules. An exact query that hits its cap without proving the position remains unresolved.
Reducing the previous exact cap from five million to one million nodes cut total exact-search work by 63.9% across the same 500 games. On 40 cold-cache positions, local p95 latency fell from 274.6 to 100.5 ms, while median latency rose from 64.5 to 88.5 ms. More positions reached the later planning stages, so lowering one budget did not make every move faster.
5. Measure each player separately
| Playing mode | Custom Expert | Fairy Hard | Games per opponent |
|---|---|---|---|
| Raw neural policy | 29.2% | 19.2% | 60 |
| Network + 256-simulation MCTS | 76.5% | 36% | 100 |
| Network + native proof/search controller | 87.5% | 70% | 100 |
Different evaluation banks and sample sizes. These are not paired estimates of the gain from adding each search component, and they are not browser scores.
Draws receive half credit: a hypothetical 100-game run with 60 wins and 20 draws would score (60 + 10) / 100 = 70%, despite winning only 60% of games. This distinction matters when interpreting a reported “score.”
Full-controller confirmation used 50 unique 4–12-ply starting boards per opponent, swapping seats for 100 games. Parent and candidate played matching starts at declared budgets. The selected controller’s 70% Hard score had a 95% interval of 63.5%–76.5%; the paired gain over the previous controller was +4.5 points with interval −3 to +12. That does not establish a clear Hard strength improvement. The smaller-budget selection passed its comparable-strength criteria.
Raw inference also passed 400 / 400 fresh immediate wins and 400 / 400 safe blocks. Tactical correctness is useful evidence, but it is only one part of strategy: the same raw network still had weak full-game scores. Separate tests answer separate questions.
6. Learn from the candidates we rejected
Better value loss did not establish better play
A shared-feature trial trained the trunk and both heads from stronger offline planning. An early recipe repeated a small set too heavily: training loss improved while validation error worsened. That is an overfitting signal—the model gets better at fitting its familiar data without improving on held-out positions.
We expanded teaching with 8,192 positions unseen by the parent, proving 7,945, and reduced early-position repetition. A frozen candidate reduced held-out value MSE from about 0.516 to 0.434, a 15.9% improvement. Outcome classification barely changed, from 71.9% to 72.2%. The errors became smaller on average without establishing a substantially better separation of outcomes.
At a 500,000-node cap, that candidate scored 71% against Hard, versus 70.5% for the champion at one million nodes. But the unchanged champion at the same 500,000-node cap scored 72.5%. The candidate failed the predeclared comparison. Evaluating only loss, or comparing different budgets without the equal-budget control, would have given a misleading conclusion.
A stronger policy teacher produced an uncertain gain
The final trial reused 768 strategic positions. Bounded deeper search completed nominal depth 14 on 543, depth 12 on 223 and depth 10 on two. Additional exact queries classified every legal action on 530 positions. Unknown actions stayed unlabelled; guaranteed optimal actions and terminal proofs took priority over heuristic preferences.
Two variants used teacher-score temperatures 80 and 240. A lower temperature concentrates probability more strongly on the best-scored action; a higher one spreads teaching probability more broadly. Thirty epochs updated only the policy projection/head, leaving shared features, tactical coefficients and the value function unchanged. This isolated a policy-target hypothesis rather than changing the whole system at once.
The softer candidate was frozen before a fresh paired confirmation at identical one-million-node and 256-simulation budgets. Hard rose from 75% to 77% on that round’s bank: +2 points with a paired 95% interval of −4.5 to +8.5 points. The interval includes a loss, and the result missed the promotion rule.
7. Decide when another run is worthwhile
The final promotion rule required at least a three-percentage-point Hard gain with a positive lower paired 95% bound, perfect fresh tactics, and no mean regression greater than five points against any reference. The candidate retained tactics and passed the mean guards, but failed the Hard requirement.
After three consecutive hypotheses produced no accepted gain, automatic v1 training stopped. The accepted checkpoint stayed unchanged. We did not repeatedly rerun confirmation until a favorable result appeared. Restarting research requires a specific hypothesis, a measurable target and a bounded allowance.
The transferable lesson is to choose the decision rule before seeing the result. Loss helps diagnose learning; playing strength at a declared runtime budget decides whether this player improves. Preserved rejected candidates explain the decision and prevent the story from becoming a collection of favorable scores.
8. Verify the whole pipeline
In the accepted continuation, all 6,716 completed games and 196,541 transitions matched both live TypeScript game engines. Audits checked legal targets, tactical labels, proof receipts, reflections and zero reserved-position overlap. Correct labels and rules matter as much as an optimizer: a network can learn a broken simulator very well.
The 288,371-byte ONNX export matched 1,000 / 1,000 actions, with individual and dynamic-batch errors below 10−4. Only the network is exported. The native proof controller, its roughly 80 MiB cache, external legal masking and browser worker behavior are separate integration work.
Browser qualification still requires encoder/controller parity, legal actions, bounded response times, cancellation of stale work and mobile testing. Final seeds 900000–999999 remain sealed, and the continuation runs share one parent. These local measurements are not independent-seed or live-browser qualification.
Model identity and reproducibility
Selected spatial continuation · training seed 193 · exact cap 1,000,000 · 256 PUCT simulations. Checkpoint SHA-256:
17d089780bc295fcc91b70ee50efa56659e8495c1004dc78d6692b02a310634fSource workflows and saved recipes live in experiments/rl/connect-four. Reproduction needs local checkpoints, teaching data and prepared native teachers; they are not all bundled in a source-only checkout. Use fresh output paths and preserve the frozen evidence.
Source reports
- Selected model, training recipe and controller evidence
- Shared-feature trial and larger MCTS comparison
- Policy-teaching trial and stopping decision
The reports link the research that informed these methods, including Expert Iteration and archived-root exploration. Our small local experiments use their own data, budgets and acceptance rules; they do not reproduce the full published benchmarks.
