PokéDreamer v3
Diffusion World Model + Dream-Trained Agent

v3 is the step this project has been building toward: a DIAMOND-style diffusion world model of Pokémon Red, and an agent whose navigation policies were trained entirely inside that model's imagination and then tested in the real game. It also introduces a measurement method: Pokémon Red's RAM gives exact ground truth (map, tile position, battle state), so imagination can be scored in game units (tiles, maps), not just pixels.

Units: a step is one button press followed by 24 Game Boy frames (≈0.4 s, the time to walk one tile). 32 steps ≈ 13 s of gameplay. These are not video frames.

Showcase video

Final results (measured)

The headline numbers for v3: how reliably the dream-trained walking works in the real game, how robust it is to where it starts, whether battles could be learned the same way (no), and whether a local LLM can replace the rule-based planner. All results are measured against the emulator's RAM as ground truth.

The badge quest

The gym quest asks whether the System 1/2 agent (learned walking, rule-based battles and goal choice) can earn the Boulder Badge (defeating Gym Leader Brock) in the real Game Boy emulator with no human input, starting from Charmander L8 in Viridian City. The agent operates under a System 1 / System 2 architecture: System 2 serves as the high-level planner over Game Boy RAM status (sequencing grinding, healing, route traversal, and gym challenges), while System 1 executes low-level navigation and battle moves. In the headline configuration with all 10 navigation legs driven by policies trained entirely inside the diffusion world model's imagination, the agent achieves 30/30 badges across three policy-training seeds (10 quests each, quest seeds 100-109), with 14 of 30 quests completing the full journey from Viridian City through Brock without triggering privileged pathfinding fallback anywhere. Seven of the ten navigation legs achieve 100/100 success across all three seeds; the only legs not reaching 100/100 on all seeds are to_pewter_pc (100 / 0 / 100, where seed 1 enters a local optimum on the Center door tile) and forest (100 / 69 / 97, where seed 1 stalls at an Antidote item ball that training data barely covers). See v3_gym.md.

Game harness: local LLMs as System 2

The game harness cleanly separates high-level planning from low-level execution: System 2 selects the current sub-goal from Game Boy RAM state (e.g., when to grind, heal, travel, or enter the gym), can call tools including Pokédex lookup and guide search over Bulbapedia (lab/agent/web_search.py), and validates every decision against game memory before dispatching to System 1. In the bake-off below the planner is the only variable: navigation is the map-reading BFS planner and battles are rules for every row, so the badge rate measures planning on top of a fixed, non-learned System 1.

Key takeaways:

• gemma4:31b is the cleanest planner: Every one of its 89 decisions passed validation without a single rejection, earning 10/10 badges with only 3 blackouts and 7,239 median presses vs 8,508.5 for the rules (10 runs each, no confidence intervals).
• gemma4:12b is the practical candidate for planner + world model on one GPU: It achieves 10/10 badges in 8,145 presses with a 10.2 GB peak GPU memory footprint, which would leave room for the world model on a 24 GB card (not tested together).

Battles: measured, and out of reach for this model

To determine whether Pokémon battle policies could also be trained purely inside the diffusion world model (Path B), wm31 was evaluated on held-out battle windows against RAM ground truth. On the full sample of 8,913 eligible held-out battle windows (29 episodes), the empirical measurements demonstrated:

• Battles end early: In 74.6% of held-out windows, the imagined battle ends more than 3 steps before the real one (compared to only 1.1% for the frame probe on real frames). A battle policy trained in this dream would learn that encounters terminate spontaneously within ~9 steps regardless of button choices.
• Battle screens degrade below copy-last from step k=2: Imagined battle frames lose to the trivial copy-last baseline for all horizons k ≥ 2 (pixel MSE 0.0629 vs 0.0556 at k=2 with non-overlapping 95% bootstrap CIs; 0.2099 vs 0.1431 at k=32), showing that a 4-frame visual context fails to preserve battle state like HP bars and menus.
• Onset is roughly right, but duration is wrong: The world model initiates battles in 53–74% of windows when real encounters occur on grass routes, but because battle frames look visually similar across turns, the 4-frame window cannot track battle progress over time.

Full analysis and architectural options are detailed in v3_battle_fidelity.md.

Seen in colour

The native-resolution upsampler (0.72M parameters) restores 72×80 greyscale frames back to native 144×160 4-shade Game Boy frames with 99.980% pixel accuracy on held-out in-distribution test episodes (test_id, with 91.4% of frames 100% exact); it functions purely as a display post-processor and is not used by any evaluation metric (upsampler.md).

The stage-2 learned colorizer (1.91M parameters) predicts per-pixel colour classes for 144×160 frames with 99.27% pixel class accuracy on held-out save states, serving strictly for visualization in figures and demo videos without any evaluation metric depending on it (colorizer.md).

Compute

All v3 training ran on a single desktop GPU (RTX 4090) and its CPU. Training time tallies every logged v3 training run, ablations and failed recipes included (76 GPU runs total, 50.0 GPU-hours); the shipped world model took 3.6 GPU-hours and each navigation skill 14.5–48.1 minutes (compute.md).

Training time (all v3 experiments, including ablations and failed recipes)

What the shipped pipeline cost

Phase 1 results (2026-11-01, measured)

An agent trained only inside the world model leaves the house in the real game. Task: start in Red's bedroom and reach Pallet Town (bedroom stairs, then the 1st floor, then the front door). Real game (PyBoy), 60-press cap:

The shortest route is ≈22 presses. The RL policy never interacted with the emulator during training: every training step was imagined by the diffusion model. It is 8 presses faster than the greedy rule (bootstrap 95% CI [-9, -6]) and statistically tied with the planner, at a fraction of the cost.

1. Diffusion world model

v1/v2 rollouts degraded after 2-5 steps (v2's latent collapsed). v3 frames stay sharp and coherent for 32+ steps.

2. RAM-grounded evaluation (held-out episodes)

A CNN probe reads map/tile from a frame (0.40-tile error on real held-out frames), so imagined frames can be scored against the true RAM. 512 windows, 95% CIs by episode-level bootstrap.

Findings.

1. Pixel metrics overstate world-model quality. By pixel error the model wins at every horizon; by RAM position it loses to "nothing moves" from ~8 steps.
2. The main failure is map identity, not drift within a map. It "teleports" to other rooms over long horizons (2% → 41% wrong map), because 4 frames of context cannot remember where you are.
3. Short-range dynamics are learned: when the player moves, the k=1 position error is 0.22 tiles vs 1.00 for "nothing moves".

3. Ground truth from the real game

The emulator harness reproduces the recorded data exactly: frames after 30 random steps are pixel-identical to the dataset (mean absolute difference 0.0000). Pressing each button from 64 real game states, in the emulator and in the model:

• One press: imagined position error 0.14-0.30 tiles (left/right/down); "up" is weaker (0.94).
• After 8 presses, right/down/up displacement matches the real game within CIs; left under-moves.
• Flaw: with no button pressed, the real player stays put, but the imagined player drifts ~1 tile per step.

4. Ablation: what fixed the 2-5 step collapse?

Same recipe with and without GameNGen-style context-noise augmentation, step-matched at 50,200 steps (single seed). Augmentation is not what fixed it: without it the model is equally stable (pixel MSE at k=32: 0.094 vs 0.129 with augmentation) and clearly better on the held-out start location (map accuracy at k=32: 0.73 vs 0.47, non-overlapping CIs). Doubling training time also did not clearly improve the RAM metrics. The stability comes from the diffusion recipe itself; the long-horizon errors point to the memory limit, not under-training.

5. Negative result: the planner exploited the model

The first planner rewarded any imagined map change. In the real game it pressed left into the bedroom wall for 60 steps: for those buttons the model imagines teleporting downstairs or outside. Once imagined map changes were credited only next to a real exit tile (and the horizon cut to 4 steps), the same model and task went from 0/1 to 21/21. The RL reward uses the same safeguard. A learned simulator must be queried with protections against its measured hallucinations.

6. RAM-conditioned hybrid model (first result)

To attack the map-identity failure at its source, a hybrid model adds a small symbolic state model (1.3M params) that predicts the next RAM state (map, tile, facing, battle) from the last 4 states and buttons, and conditions the diffusion renderer on that predicted state. Trained step-matched to the pixel-only model without augmentation (50,200 steps, no context noise; single seed), evaluated on the same 512 held-out windows. It starts from the true RAM of the 4 context frames, which is always available in the real game.

Reading: the symbolic state model solves state tracking: it almost never loses the map and beats "nothing moves" on position at every horizon, which confirms that the failure was memory, not pixels. The renderer only partly follows the state it is given: rendered frames still show the wrong map 26% of the time at 32 steps (CIs overlap with the pixel-only model at this sample size). Making the renderer obey its conditioning is the next step.

Status at completion (2026-11-07)

Summary of planned v3 items at project completion:

Phase 2 plan (as written 2026-11-01)

Outcome: step 1 (classifier-free guidance on the hybrid) was tried and was negative (v3_dwm_eval.md); step 2 done (v3_agent.md, v3_gym.md); step 3 (the agent on the hybrid) not done; step 4 (rewards without hand-given exits) not done; step 5 done for an LLM over RAM state, without screenshots (v3_system2.md).

Ordered; each step has a measurable success criterion, judged in the real game where possible.

1. Make the renderer obey its state. Fine-tune the hybrid model with classifier-free guidance on the RAM conditioning (train with the state dropped ~15% of the time; at sampling, push the prediction toward the conditioned one). Success: rendered wrong-map rate at 32 steps well below the current 25.9%, approaching the state model's 1.1%, without losing pixel quality.
2. Harder tasks from a starter-Pokémon save state. The save state is generated by script from the ROM (intro, Oak's lab, choose a starter, rival battle), so it is reproducible without downloading game data. Tasks: reach Route 1 from Pallet Town (Oak no longer stops you), then reach Viridian City through tall grass, wild battles and ledges. Success: zero-shot real-game success rate of a policy trained in imagination vs greedy and random baselines, as in Phase 1.
3. Agent on the hybrid model. Retrain the imagination policy inside the guided hybrid model and compare real-game success and route length with the pixel-only model on the same tasks.
4. Reward without hand-given exits. Replace exit-tile rewards with goal maps / goal images so the System 1 policy is goal-conditioned.
5. System 2. A local LLM/VLM planner that turns RAM state + screenshot into a sequence of goal maps for System 1, scored by real-game progress (maps reached, badges).

Reproducing Phase 1

Code lives in lab/. You need your own legally obtained ROM at Pokemon - Red Version (USA, Europe).gb (not included; SHA-1 ea9bcae617fdf159b045185467ae58b2e4a48b9a).

python -m lab.data                                   # 72x80 cache + episode splits from the HF dataset
python -m lab.dwm.train --name main --minutes 120   # diffusion world model (RTX 4090: ~15 steps/s)
python -m lab.probe                                  # frame -> RAM probe
python -m lab.dwm.eval_rollouts --run main           # RAM-grounded evaluation
python -m lab.env --make-states                      # emulator save states (needs the ROM)
python -m lab.dwm.real_control --run main            # controllability vs the real game
python -m lab.agent.imagine_rl --minutes 30          # System 1 trained in imagination
python -m lab.agent.mpc --policy learned --episodes 100   # zero-shot real-game test
python -m lab.play --run main                        # play inside the world model (p = screenshot)

Long jobs checkpoint every 15 minutes and resume; creating the file work/PAUSE pauses training (checkpoint, GPU idle) until it is removed.

Caveats

• One training run per configuration (one seed); 16 in-distribution test episodes, hence wide CIs.
• The probe that scores imagined frames is trained on the same maps; it is unreliable on the held-out start location (4.8-tile error on real frames there), so those position numbers are reported as inconclusive.
• All world-model rollouts are open-loop on recorded action sequences of a PPO agent.

Original v3 plan (2026-06) and status

The original plan below is kept for reference; the status column reflects 2026-11-07.