Skip to content
← All experiments

Teaching Snake to take the long view.

How a small neural policy learned to fill the board with fewer movements: expert demonstrations, spatial features, guided search and careful evaluation.

Snake · Research recorded 7–8 October 2026

Concept illustration · measured evidence below

1. Define the task before training

Finding the next apple is a short-term problem. Completing Snake is a long-term one: a move that gets food quickly can leave the head trapped behind its own body thousands of movements later. Our question was whether a small neural policy could fill every playable cell while taking fewer movements than a safe cycle-following baseline.

The board is 18 × 18 and the snake starts with three occupied cells. Open garden has 324 playable cells, The long path has 316, and Twin hedges has 312. A complete run therefore collects 321, 313 or 309 apples respectively. Walls do not count toward completion; a win ends with the entire playable board occupied and no food remaining.

An earlier experiment rewarded a small apple target. That was a useful beginner task, but it measured a different objective. We corrected the full-board task before optimizing efficiency. Otherwise, a model could appear successful while never learning to finish the game.

2. Watch the frozen model

These are verified recordings of the selected policy, rather than live neural inference. Choose a course, pause, step through individual movements, or jump to the complete board. Each recording uses the first seed of its fresh course stream; we did not search for the fastest demonstration.

18 × 18 · green snake · rust apple · grey walls

Watch the board fill

Every move comes from the frozen model’s verified recording. Playback uses the same rules as our Snake game.

Recorded seed 1,900,000 · 324 playable cells

Loading and checking recording…

Recorded demonstration. This page does not run live neural inference or enter human high scores.

Replay executes recorded actions through the website’s game rules. Changing playback speed changes how quickly you watch, not the model’s decisions. The player writes no Snake saves or high scores.

3. Give the network the right information

A policy maps the visible game state to an action. Here the actions are relative to the current heading: straight, left and right. The network outputs three logits—scores before any probability conversion—and the evaluated policy chooses the largest. Its 10,694 learned parameters are the weights adjusted during training.

Three complementary inputs

Spatial board · 9 × 18 × 18
Nine aligned grids describe four body directions toward the head, the head, tail, food, walls and ordered body. Convolutions read nearby patterns, such as food beside the head or a body segment blocking a route.
Context · 8 values
Four heading flags, three course flags and the fraction of the board occupied tell the network which way it faces, which course it is on and how crowded the board has become.
Candidate geometry · 3 × 18 values
Each possible action receives features about immediate legality, cyclic body order, food and tail distance, growth room and progress along a certified route. These are engineered from visible state; the model does not receive future apples or the hidden random-generator state.

A Hamiltonian cycle visits every playable cell once before returning to its start. Following a suitable cycle gives the snake a safe ordering to preserve. A shortcut can reduce travel, but must leave enough room for the tail and future growth. That geometric knowledge is supplied by our encoder and expert; it was not discovered entirely from random play.

A small spatial network with action-specific scores

Two 3 × 3 convolution layers use 9→8→8 channels, followed by pooling to a 3 × 3 representation and a 72→24 projection. Adding the eight context values produces a 32-value context. For each action, a learned linear geometry score is combined with a bounded nonlinear residual from 50→64→32→1 layers. A separate 32→64→3 value head predicts completion, discounted return and remaining movements.

4. Learn from an expert, then refine

  1. Expert routes18 complete games
  2. Warm-start35 supervised epochs
  3. Guided improvement9 games with search
  4. Freeze & compareWin first, then speed
Training uses an expert and search. The evaluated actor chooses its own raw neural actions.

Stage A · imitation gives a useful starting point

The shortcut expert preserves cyclic body order and growth space, and stops taking shortcuts above half board fill. Its 18 full games produced 264,460 movements and 73,744 sampled teaching records, balanced by course, body phase and steering action. This warm-start teaches the network useful behavior before asking it to improve through experience.

We trained for 35 epochs with Adam, learning rate 0.003, batches of 256 and gradient clipping at 5. An epoch is a pass through the supplied training records; a batch is the group used for one gradient update. Gradient clipping limits unusually large updates. These settings describe the measured recipe, rather than universal best choices.

What the losses ask the network to learn

Policy cross-entropy penalizes putting too little probability on the teacher’s action. Binary cross-entropy trains the completion prediction. SmoothL1 losses fit discounted return and remaining movement targets; SmoothL1 is quadratic near a match and less sensitive to large errors than a purely squared loss. The recipe weights movement loss by 5 and admissibility-margin regularization by 3.

The margin term encourages admissible route choices to outrank other choices through learned scores. After training, the selected weights had a checked dominance margin of 0.590878 under the declared feature bounds. This is conditional on an admissible candidate existing; it does not prove recovery from arbitrary positions.

Stage B · use search to produce improved targets

Nine guided games supply policy/value improvement data. Every three games, the latest 40,000 records receive five training epochs at learning rate 0.0005, with Adam recreated for the phase. At movement multiples of 128, where several admissible actions exist, PUCT search runs eight simulations with exploration coefficient 0.5 and depth 24 movement ticks. Forced sequences can extend up to 32 ticks.

PUCT balances branches that look promising with branches that have received less exploration. Its visit distribution becomes a teaching target: search effort indicates which choices merit probability. Search samples future apples uniformly from free cells instead of reading the real hidden seed. Every simulated movement still counts toward the budget and discount exponent.

Reward and a worked discount example

Rewards are 1 / (capacity − 3) per apple, +5 for completion, −5 for collision and −0.00001 per movement, with discount 0.9999. On Open garden, one apple gives about 0.00312. A future reward R arriving k movements later contributes 0.9999k × R to discounted return. A 32-movement forced route discounts by 0.999932, not just one discount because the search compressed it into one branch.

Training restricts rollout/search actions to the certified cyclic-order subset; evaluation does not. That difference matters when interpreting what the learning system has actually demonstrated. The measured bounded experiment used 402,885 expert, real-policy and search movements in 416.33 seconds, within its declared 1,800,000-movement and 1,000-second caps.

5. Compare on matching starts

The frozen selected policy completed 120 / 120 full-board starts: an existing 60-start suite and a separate fresh 60-start suite, with 20 starts per course in each. Neither suite contained deaths, inactivity cutoffs, movement cutoffs or assisted runs.

Fresh validation · 20 matching starting seeds per course
CourseCycle mean movesNeural mean movesReduction
Open garden25,815.3015,436.5040.20%
The long path24,784.1514,788.9540.33%
Twin hedges23,941.5514,455.2539.62%

Across fresh starts, the mean fell from 24,847.00 to 14,893.57 movements. The calculation is (24,847.00 − 14,893.57) / 24,847.00 × 100 = 40.06% fewer movements. The paired, course-stratified bootstrap 95% interval is 39.31%–40.79%.

That interval measures starting-seed variability for this fixed trained model. It does not measure variation across independently trained models. Matching initial seeds also does not guarantee identical later apple locations: different paths change which cells are free.

A bootstrap repeatedly resamples the observed comparisons to estimate variability. Pairing keeps the two policies’ matching starts together, while stratification keeps courses represented. This is more informative than comparing two unrelated batches of games, but it remains finite validation on three fixed course geometries.

6. Separate a result from its cause

The warm-start and three refinement checkpoints all won 15 / 15 selection starts with identical movement statistics. The explicit tie rule selected the earliest post-refinement checkpoint. The later updates changed weights, but the observed efficiency improvement was already present after imitation.

The history explains why representation mattered. The earlier full-board v2 policy followed a route using head, heading and course features, without food/body-dependent shortcut choices. It could finish reliably but lacked the information needed to adapt its path efficiently. Adding spatial state and candidate geometry made the relevant distinctions available to the learner.

Three practical lessons carry over to other ML work: define the real task before optimizing; make relevant information visible to the model; and compare the warm-start with the refined model before attributing improvement to the newest technique.

7. Check the environment and the export

The three public recordings contain 43,508 initial/resulting states, all checked against the website’s TypeScript rules. They finish with 324, 316 and 312 occupied cells and no food. Download hashes are checked before playback. A plausible animation alone would not establish correct game behavior.

The portable ONNX network is 47,587 bytes and matched 1,000 / 1,000 unmasked actions from the authoritative trained policy. Maximum logit error was 2.38 × 10−6, and value error 3.81 × 10−6. ONNX carries the learned graph; geometry calculations remain outside it and must also match before live browser inference can be qualified.

Training seed 47 represents one lineage. Independent training seeds, arbitrary saved-state recovery, resized boards, new obstacles and calibrated off-domain win probabilities remain unestablished. Final-test seeds at or above 2,000,000 stay reserved. Browser/mobile encoder parity and response times are separate next steps.

Model identity and reproducibility

Movement-efficient v3 · seed 47 · selected copy of round-001.pt. Checkpoint SHA-256:

1a99a8b938fd15cf07071c9c6708b3e650a4726fe1f010629d4fe7cd15dabccd

The source workflow lives in experiments/rl/snake. Reproduction requires prepared local dependencies and artifacts, and a fresh output directory. Reports provide the training, selection, evaluation and export recipe. A rerun is not guaranteed to produce identical weights or timing.

Source reports

The model card links the research that informed the approach, including AlphaSnake and the cycle/shortcut algorithm. Our board sizes, engineered inputs and training budget differ; these results do not reproduce those papers’ full benchmarks.