CS2 MIRA-style world model: codec + WM weights, streaming code, model card
Browse files- .gitattributes +3 -0
- README.md +131 -0
- assets/codec_reconstruction.png +3 -0
- assets/real_vs_generated_final.png +3 -0
- assets/walkthrough.gif +3 -0
- codec_069000.pt +3 -0
- configs/actions/cs2.yaml +16 -0
- modal_app.py +103 -0
- patches/dino_hf_backbone.patch +60 -0
- scripts/consolidate_codec.py +49 -0
- scripts/eval_cs2_codec.py +61 -0
- scripts/train_cs2_codec.py +128 -0
- scripts/train_cs2_wm.py +121 -0
- src/mira/data/cs2_stream.py +353 -0
- world_model_030000.pt +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
assets/codec_reconstruction.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
assets/real_vs_generated_final.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
assets/walkthrough.gif filter=lfs diff=lfs merge=lfs -text
|
README.md
ADDED
|
@@ -0,0 +1,131 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
tags:
|
| 4 |
+
- world-model
|
| 5 |
+
- video-generation
|
| 6 |
+
- diffusion
|
| 7 |
+
- flow-matching
|
| 8 |
+
- counter-strike
|
| 9 |
+
- reinforcement-learning-environment
|
| 10 |
+
library_name: pytorch
|
| 11 |
+
---
|
| 12 |
+
|
| 13 |
+
# CS2 Latent World Model (MIRA-style)
|
| 14 |
+
|
| 15 |
+
An **action-conditioned world model for Counter-Strike 2** — feed it keyboard + mouse actions and it
|
| 16 |
+
imagines the next frames. Built by adapting [MIRA](https://github.com/mira-wm/mira) to stream
|
| 17 |
+
[RekaAI/CS2-10k](https://huggingface.co/datasets/RekaAI/CS2-10k) (63 TB) with **no local dataset copy**.
|
| 18 |
+
|
| 19 |
+
<p align="center">
|
| 20 |
+
<img src="assets/walkthrough.gif" alt="Walking around inside the world model" width="640"/>
|
| 21 |
+
</p>
|
| 22 |
+
|
| 23 |
+
*Driving the world model with "hold W + sweep the mouse" — every frame above is **generated**, not real gameplay.*
|
| 24 |
+
|
| 25 |
+
> **Status: research prototype.** It works end-to-end (coherent, action-responsive rollouts) but is
|
| 26 |
+
> small, blurry, and trained on very little data. Limitations are documented honestly below.
|
| 27 |
+
|
| 28 |
+
## Real vs. generated
|
| 29 |
+
|
| 30 |
+
| Real gameplay (top) vs. world model (bottom) |
|
| 31 |
+
|---|
|
| 32 |
+
|  |
|
| 33 |
+
|
| 34 |
+
| Codec reconstruction — ground truth ‖ reconstruction |
|
| 35 |
+
|---|
|
| 36 |
+
|  |
|
| 37 |
+
|
| 38 |
+
## Architecture
|
| 39 |
+
|
| 40 |
+
**Stage 1 — Video codec (RAEv2)**
|
| 41 |
+
- Frozen **DINOv3-L/16** (`facebook/dinov3-vitl16-pretrain-lvd1689m`) backbone, layer aggregation
|
| 42 |
+
over blocks `[11,13,15,17,19,21,23]`.
|
| 43 |
+
- Strided-conv bottleneck → **32-channel latent at /32 spatial, /2 temporal**.
|
| 44 |
+
- ViT decoder (width 512, depth 6, patch 16). ~331 M params, mostly frozen.
|
| 45 |
+
|
| 46 |
+
**Stage 2 — Latent world model**
|
| 47 |
+
- Causal **flow-matching diffusion transformer** over codec latents — **12.5 M trainable** params
|
| 48 |
+
(hidden 384, 4 layers, 6 heads).
|
| 49 |
+
- AdaLN conditioning on **per-frame keyboard + mouse** actions; diffusion forcing; action dropout 0.1.
|
| 50 |
+
- Autoregressive rollout with a streaming KV-cache.
|
| 51 |
+
|
| 52 |
+
**IO:** 288×512 @ 24 fps, 16-frame clips (CS2 is 48 fps → integer /2 downsample).
|
| 53 |
+
|
| 54 |
+
## Action space
|
| 55 |
+
|
| 56 |
+
10 keys — `W A S D J C R V [ ]` (move, jump, crouch, walk, freefall, fire, scope) as a multi-hot
|
| 57 |
+
vector, **plus continuous mouse deltas**.
|
| 58 |
+
|
| 59 |
+
> Upstream MIRA conditions on *key presses only* (Rocket League is keyboard-only). CS2 is
|
| 60 |
+
> mouse-driven, so mouse deltas are populated here and consumed by MIRA's existing `ActionEncoder`
|
| 61 |
+
> (symlog-normalized + temporally pooled).
|
| 62 |
+
|
| 63 |
+
## Files
|
| 64 |
+
|
| 65 |
+
| File | What |
|
| 66 |
+
|---|---|
|
| 67 |
+
| `codec_069000.pt` | Codec weights (decoder + bottleneck; frozen DINOv3 rebuilt from HF) |
|
| 68 |
+
| `world_model_030000.pt` | World model weights (12.5 M) |
|
| 69 |
+
| `scripts/`, `src/`, `configs/` | Training / eval / streaming code |
|
| 70 |
+
| `patches/dino_hf_backbone.patch` | Load gated DINOv3 via `transformers` instead of Meta's `.pth` |
|
| 71 |
+
| `modal_app.py` | Run the same training on a cloud GPU |
|
| 72 |
+
|
| 73 |
+
## Training
|
| 74 |
+
|
| 75 |
+
| | Codec | World model |
|
| 76 |
+
|---|---|---|
|
| 77 |
+
| Steps | 69,000 | 30,000 |
|
| 78 |
+
| Hardware | RTX 5060 Ti (16 GB) + RTX PRO 6000 (Modal) | RTX 5060 Ti |
|
| 79 |
+
| Loss | L1 reconstruction | flow-matching diffusion |
|
| 80 |
+
| Final | ema 0.1392 | ema 2.41 |
|
| 81 |
+
|
| 82 |
+
Trained on the untarred `sample/` split: **3 matches / 374 clips / 44 (match, round) groups.**
|
| 83 |
+
|
| 84 |
+
## Results
|
| 85 |
+
|
| 86 |
+
- **Codec reconstruction: 24.75 dB PSNR** (vs 19.7 dB with a random backbone). Recovers room
|
| 87 |
+
geometry and recognizable player figures; no fine texture.
|
| 88 |
+
- **Rollouts stay coherent** — players persist across the clip, room geometry is stable.
|
| 89 |
+
- **Controllability** — mean abs pixel difference of a generated rollout vs an idle-action rollout:
|
| 90 |
+
- `forward` vs `idle`: **0.025 – 0.037**
|
| 91 |
+
- `turn` vs `idle`: **0.040**
|
| 92 |
+
- Non-zero **and growing over the clip** → the model integrates actions over time rather than
|
| 93 |
+
ignoring them. This was the go/no-go test, and it passed.
|
| 94 |
+
|
| 95 |
+
## Limitations
|
| 96 |
+
|
| 97 |
+
- **Blurry.** The /32, 32-channel latent + a depth-6 decoder caps detail (no faces, weapon detail,
|
| 98 |
+
floor texture). The paper uses a ViT-XL decoder.
|
| 99 |
+
- **Long-horizon drift.** Coherent for several frames past context, then degrades — dissolving to
|
| 100 |
+
generic wall texture or darkening. 16-frame clips vs the paper's 80.
|
| 101 |
+
- **Tiny dataset — 3 matches.** The dominant constraint; there's little to generalize from.
|
| 102 |
+
- **Tiny model** — 12.5 M vs the paper's 1 B. Long-horizon coherence is what capacity buys.
|
| 103 |
+
- **Single-perspective only.** MIRA's defining contribution is *multiplayer* (joint, time-aligned
|
| 104 |
+
multi-POV with per-player action attribution). Only the single-player stage is trained here. The
|
| 105 |
+
multi-POV data path exists (`cs2_stream` groups POVs by `(match_id, round_number)`, with ranged
|
| 106 |
+
tar-member fetch for the full split) but the joint stage was never trained.
|
| 107 |
+
- **Action semantics unverified.** We measure that actions *change* the output, not that "forward"
|
| 108 |
+
moves precisely forward — fidelity is too low to confirm direction visually.
|
| 109 |
+
|
| 110 |
+
## Intended use / out of scope
|
| 111 |
+
|
| 112 |
+
Research on world models and action-conditioned video prediction. **Not** a game, an anti-cheat
|
| 113 |
+
signal, a player model, or anything deployable. Do not present generated frames as real gameplay.
|
| 114 |
+
|
| 115 |
+
## Attribution & licensing
|
| 116 |
+
|
| 117 |
+
- **MIRA** — architecture/training code (upstream; this repo ships patches + new modules).
|
| 118 |
+
- **DINOv3** (Meta) — frozen backbone, **gated weights** under Meta's license (not redistributed here).
|
| 119 |
+
- **CS2-10k** (RekaAI) — **CC BY-NC 4.0**.
|
| 120 |
+
- Counter-Strike 2 imagery © Valve; generated frames derive from that footage.
|
| 121 |
+
|
| 122 |
+
**Non-commercial research use only.**
|
| 123 |
+
|
| 124 |
+
## Usage
|
| 125 |
+
|
| 126 |
+
```bash
|
| 127 |
+
export RS_DINO_HF=facebook/dinov3-vitl16-pretrain-lvd1689m # requires HF access to gated DINOv3
|
| 128 |
+
python scripts/consolidate_codec.py # pack codec_069000.pt for the world model
|
| 129 |
+
python scripts/train_cs2_wm.py --test # controllability check (generates rollouts)
|
| 130 |
+
python scripts/eval_cs2_codec.py # reconstruction PSNR + side-by-side frames
|
| 131 |
+
```
|
assets/codec_reconstruction.png
ADDED
|
Git LFS Details
|
assets/real_vs_generated_final.png
ADDED
|
Git LFS Details
|
assets/walkthrough.gif
ADDED
|
Git LFS Details
|
codec_069000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:61001017c77e1921911f88696f9fd747e69a2005c4ca8c6d91afdc56d9b99842
|
| 3 |
+
size 111627229
|
configs/actions/cs2.yaml
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Counter-Strike 2 keybinds for CS2-10k (RekaAI). Per-frame `actions` string chars, stable order.
|
| 2 |
+
# `target_fps` must divide the 48fps source (compute_stride is integer): 24, 16, 12, 8, 6.
|
| 3 |
+
# Unlike keyboard-only Rocket League, CS2 also carries continuous mouse deltas
|
| 4 |
+
# (mouse_x_delta/mouse_y_delta), populated into ActionTensors.mouse_movements by cs2_stream.
|
| 5 |
+
target_fps: 24
|
| 6 |
+
valid_keys:
|
| 7 |
+
- W # forward
|
| 8 |
+
- A # left
|
| 9 |
+
- S # back
|
| 10 |
+
- D # right
|
| 11 |
+
- J # jump
|
| 12 |
+
- C # crouch
|
| 13 |
+
- R # walk (slow)
|
| 14 |
+
- V # freefall / release
|
| 15 |
+
- "[" # fire (attack)
|
| 16 |
+
- "]" # scope / secondary
|
modal_app.py
ADDED
|
@@ -0,0 +1,103 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Run the CS2 MIRA training on Modal (e.g. an RTX PRO 6000 / H200, 90GB+ VRAM) to finish faster.
|
| 2 |
+
|
| 3 |
+
Same model config as local (nothing upgraded) — just a much faster GPU and no thermal pauses.
|
| 4 |
+
Checkpoints live in a Modal Volume so they persist and can be pulled back down.
|
| 5 |
+
|
| 6 |
+
SETUP (once, on your machine with the Modal CLI + `modal setup` done):
|
| 7 |
+
# 1. store your HF token (gated DINOv3 + CS2 dataset) as a Modal secret named "huggingface":
|
| 8 |
+
modal secret create huggingface HF_TOKEN=hf_xxxxxxxx
|
| 9 |
+
# 2. seed the volume with your latest local codec checkpoint so Modal RESUMES (not restarts):
|
| 10 |
+
modal volume create cs2-runs
|
| 11 |
+
modal volume put cs2-runs runs/cs2_codec_dino/codec_048000.pt cs2_codec_dino/codec_048000.pt
|
| 12 |
+
|
| 13 |
+
RUN (from the mira/ dir):
|
| 14 |
+
modal run modal_app.py # resume codec -> 120k on the cloud GPU
|
| 15 |
+
modal run modal_app.py --stage wm # (optional) train the small world model instead
|
| 16 |
+
|
| 17 |
+
PULL RESULTS BACK:
|
| 18 |
+
modal volume get cs2-runs cs2_codec_dino/codec_120000.pt ./runs/cs2_codec_dino/
|
| 19 |
+
|
| 20 |
+
Default GPU is the RTX PRO 6000 (96GB Blackwell). The config uses only ~6GB, so the VRAM is plenty;
|
| 21 |
+
raise CS2_BS to use more of it for speed. Override with MODAL_GPU=... if that string isn't accepted.
|
| 22 |
+
NOTE: RTX PRO 6000 is Blackwell (sm_120) — the image already pins torch cu132, which has sm_120
|
| 23 |
+
kernels (same reason we needed cu132 on your local 5060 Ti).
|
| 24 |
+
"""
|
| 25 |
+
import os
|
| 26 |
+
import modal
|
| 27 |
+
|
| 28 |
+
GPU = os.environ.get("MODAL_GPU", "RTX-PRO-6000") # 96GB Blackwell; override via MODAL_GPU if needed
|
| 29 |
+
|
| 30 |
+
image = (
|
| 31 |
+
modal.Image.debian_slim(python_version="3.11")
|
| 32 |
+
.apt_install("git", "ffmpeg")
|
| 33 |
+
# torch cu132 = the build with Blackwell (sm_120) kernels, matching what we validated locally;
|
| 34 |
+
# it also runs fine on Hopper (H100/H200). Swap the index if your GPU needs a different CUDA.
|
| 35 |
+
.pip_install("torch==2.13.0", "torchvision==0.28.0",
|
| 36 |
+
index_url="https://download.pytorch.org/whl/cu132")
|
| 37 |
+
.pip_install(
|
| 38 |
+
"transformers<5", "huggingface_hub", "hf_transfer", "safetensors", "av", "pyarrow", "pandas",
|
| 39 |
+
"omegaconf", "pydantic", "einops", "torchmetrics", "termcolor", "fvcore", "iopath",
|
| 40 |
+
"pillow", "numpy", "tqdm", "rich",
|
| 41 |
+
)
|
| 42 |
+
# ship our code (cs2_stream, the HF DINOv3 adapter, the train scripts, the cs2 action config)
|
| 43 |
+
.add_local_dir("src", "/root/mira/src")
|
| 44 |
+
.add_local_dir("scripts", "/root/mira/scripts")
|
| 45 |
+
.add_local_dir("configs", "/root/mira/configs")
|
| 46 |
+
)
|
| 47 |
+
|
| 48 |
+
app = modal.App("cs2-mira", image=image)
|
| 49 |
+
runs = modal.Volume.from_name("cs2-runs", create_if_missing=True)
|
| 50 |
+
COMMON = dict(volumes={"/root/mira/runs": runs},
|
| 51 |
+
secrets=[modal.Secret.from_name("huggingface")],
|
| 52 |
+
timeout=24 * 3600)
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
def _run(script: str, extra_env: dict):
|
| 56 |
+
import subprocess
|
| 57 |
+
env = {**os.environ,
|
| 58 |
+
"RS_DINO_HF": "facebook/dinov3-vitl16-pretrain-lvd1689m", # real gated weights via HF
|
| 59 |
+
"HF_HUB_ENABLE_HF_TRANSFER": "1",
|
| 60 |
+
"PYTHONIOENCODING": "utf-8",
|
| 61 |
+
"PYTHONUNBUFFERED": "1",
|
| 62 |
+
**extra_env}
|
| 63 |
+
subprocess.run(["python", "-u", script], cwd="/root/mira", env=env, check=True)
|
| 64 |
+
runs.commit() # persist checkpoints written to the volume
|
| 65 |
+
|
| 66 |
+
|
| 67 |
+
@app.function(gpu=GPU, **COMMON)
|
| 68 |
+
def train_codec():
|
| 69 |
+
# identical config to local: T=16, 288x512@24fps, small decoder. Resumes from newest checkpoint
|
| 70 |
+
# already in the volume (upload your local one first). No thermal pauses on the cloud GPU.
|
| 71 |
+
_run("scripts/train_cs2_codec.py", {
|
| 72 |
+
"CS2_T": "16",
|
| 73 |
+
# NOTE: batch size does NOT reduce steps-to-finish — 150k steps @ bs64 is ~64x the compute of
|
| 74 |
+
# your 120k @ bs1 plan. To finish the SAME run faster, keep bs small and let the fast GPU do
|
| 75 |
+
# it. bs4 + workers keeps the decode pipeline fed without ballooning per-step cost.
|
| 76 |
+
"CS2_BS": os.environ.get("CS2_BS", "40"),
|
| 77 |
+
"CS2_WORKERS": os.environ.get("CS2_WORKERS", "8"), # Linux: parallel clip decode
|
| 78 |
+
"CS2_STEPS": os.environ.get("CS2_STEPS", "120000"),
|
| 79 |
+
"CS2_KEEP": "3",
|
| 80 |
+
"CS2_OUT": "runs/cs2_codec_dino",
|
| 81 |
+
"CS2_PAUSE_EVERY": "0",
|
| 82 |
+
})
|
| 83 |
+
|
| 84 |
+
|
| 85 |
+
@app.function(gpu=GPU, **COMMON)
|
| 86 |
+
def train_wm():
|
| 87 |
+
# the same tiny world model, on the consolidated codec in the volume.
|
| 88 |
+
_run("scripts/train_cs2_wm.py", {
|
| 89 |
+
"CS2_T": "16",
|
| 90 |
+
"CS2_WM_STEPS": os.environ.get("CS2_WM_STEPS", "20000"),
|
| 91 |
+
"CS2_WM_OUT": "runs/cs2_wm",
|
| 92 |
+
"CS2_CODEC": "runs/cs2_codec_consolidated/codec.pth",
|
| 93 |
+
})
|
| 94 |
+
|
| 95 |
+
|
| 96 |
+
@app.local_entrypoint()
|
| 97 |
+
def main(stage: str = "codec"):
|
| 98 |
+
if stage == "codec":
|
| 99 |
+
train_codec.remote()
|
| 100 |
+
elif stage == "wm":
|
| 101 |
+
train_wm.remote()
|
| 102 |
+
else:
|
| 103 |
+
raise SystemExit("stage must be 'codec' or 'wm'")
|
patches/dino_hf_backbone.patch
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
diff --git a/src/mira/codec/dino.py b/src/mira/codec/dino.py
|
| 2 |
+
index f2fd849..c0dbf25 100644
|
| 3 |
+
--- a/src/mira/codec/dino.py
|
| 4 |
+
+++ b/src/mira/codec/dino.py
|
| 5 |
+
@@ -62,6 +62,42 @@ DEFAULT_DINO_LAYERS = {
|
| 6 |
+
}
|
| 7 |
+
|
| 8 |
+
|
| 9 |
+
+class _HFDinoBackbone(nn.Module):
|
| 10 |
+
+ """Adapter exposing the hub `get_intermediate_layers` API over a HuggingFace `transformers`
|
| 11 |
+
+ DINOv3 model. Lets us use the real gated weights (e.g. facebook/dinov3-vitl16-pretrain-lvd1689m,
|
| 12 |
+
+ which ship as HF safetensors) without the Meta `.pth` that torch.hub expects. Enabled by setting
|
| 13 |
+
+ the env var RS_DINO_HF to the HF repo id."""
|
| 14 |
+
+
|
| 15 |
+
+ def __init__(self, repo_id: str):
|
| 16 |
+
+ super().__init__()
|
| 17 |
+
+ from transformers import AutoModel
|
| 18 |
+
+ self.model = AutoModel.from_pretrained(repo_id).eval()
|
| 19 |
+
+ for p in self.model.parameters():
|
| 20 |
+
+ p.requires_grad_(False)
|
| 21 |
+
+ cfg = self.model.config
|
| 22 |
+
+ self.n_prefix = 1 + int(getattr(cfg, "num_register_tokens", 0)) # cls + register tokens
|
| 23 |
+
+ self.patch = int(getattr(cfg, "patch_size", 16))
|
| 24 |
+
+ self.final_norm = getattr(self.model, "layernorm", None) or getattr(self.model, "norm", None)
|
| 25 |
+
+
|
| 26 |
+
+ @torch.no_grad()
|
| 27 |
+
+ def get_intermediate_layers(self, x, n, norm: bool = True, reshape: bool = True):
|
| 28 |
+
+ out = self.model(pixel_values=x, output_hidden_states=True)
|
| 29 |
+
+ hs = out.hidden_states # (num_blocks + 1) tensors, index 0 = embeddings
|
| 30 |
+
+ n_blocks = len(hs) - 1
|
| 31 |
+
+ idxs = tuple(n) if isinstance(n, (list, tuple)) else tuple(range(n_blocks - int(n), n_blocks))
|
| 32 |
+
+ H, W = x.shape[-2] // self.patch, x.shape[-1] // self.patch
|
| 33 |
+
+ feats = []
|
| 34 |
+
+ for j in idxs:
|
| 35 |
+
+ h = hs[j + 1] # output after block j
|
| 36 |
+
+ if norm and self.final_norm is not None:
|
| 37 |
+
+ h = self.final_norm(h)
|
| 38 |
+
+ patch = h[:, self.n_prefix:, :] # drop cls + register tokens
|
| 39 |
+
+ if reshape:
|
| 40 |
+
+ patch = patch.reshape(patch.shape[0], H, W, patch.shape[-1]).permute(0, 3, 1, 2).contiguous()
|
| 41 |
+
+ feats.append(patch)
|
| 42 |
+
+ return feats
|
| 43 |
+
+
|
| 44 |
+
+
|
| 45 |
+
class DinoModel(nn.Module):
|
| 46 |
+
# The hub-loaded (or preloaded) backbone; typed loosely as its API is dynamic.
|
| 47 |
+
dino_model: Any
|
| 48 |
+
@@ -90,8 +126,12 @@ class DinoModel(nn.Module):
|
| 49 |
+
self.layers = DEFAULT_DINO_LAYERS[dino_model]
|
| 50 |
+
|
| 51 |
+
self.dino_dim = DINO_DIM[dino_model]
|
| 52 |
+
+ _hf_repo = os.environ.get("RS_DINO_HF")
|
| 53 |
+
if preloaded_dino_module:
|
| 54 |
+
self.dino_model = preloaded_dino_module
|
| 55 |
+
+ elif _hf_repo:
|
| 56 |
+
+ logger.info(f"Loading DINOv3 via HuggingFace transformers: {_hf_repo}")
|
| 57 |
+
+ self.dino_model = _HFDinoBackbone(_hf_repo)
|
| 58 |
+
else:
|
| 59 |
+
logging.getLogger("dinov3").setLevel(logging.WARNING) # suppress noisy dinov3 logging
|
| 60 |
+
logger.info(f"Loading DINOv3 model, variant {dino_model}, {compile=}")
|
scripts/consolidate_codec.py
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Consolidate our decoder-only CS2 codec checkpoint into the format MIRA's world model expects:
|
| 2 |
+
a full VideoCodec state_dict (real HF DINOv3 backbone + our trained decoder) + a codec_config.yaml
|
| 3 |
+
alongside, so LatentWorldModel can load it via VideoCodec.load_from_checkpoint.
|
| 4 |
+
"""
|
| 5 |
+
import os, sys
|
| 6 |
+
os.environ.setdefault("RS_DINO_HF", "facebook/dinov3-vitl16-pretrain-lvd1689m")
|
| 7 |
+
sys.path.insert(0, "src")
|
| 8 |
+
import torch
|
| 9 |
+
from omegaconf import OmegaConf
|
| 10 |
+
from mira.codec import VideoCodec
|
| 11 |
+
from mira.codec.config import (
|
| 12 |
+
VideoCodecConfig, RAEEncoderConfig, ViTDecoderConfig, StridedConvBottleneckConfig, ImageConfig,
|
| 13 |
+
)
|
| 14 |
+
|
| 15 |
+
SRC = os.environ.get("CS2_CODEC_CKPT", "runs/cs2_codec_dino/codec_036000.pt")
|
| 16 |
+
OUT = os.environ.get("CS2_CODEC_OUT", "runs/cs2_codec_consolidated")
|
| 17 |
+
T, H, W = int(os.environ.get("CS2_T", "16")), 288, 512
|
| 18 |
+
|
| 19 |
+
img = ImageConfig(height=H, width=W, channels=3, timesteps=T, fps=24)
|
| 20 |
+
enc = RAEEncoderConfig(latent_dim=32, rae_model="dinov3_vitl16",
|
| 21 |
+
aggregation_layers=[11, 13, 15, 17, 19, 21, 23],
|
| 22 |
+
bottleneck=StridedConvBottleneckConfig(stride=2, temporal_stride=2, noise_tau=0.0),
|
| 23 |
+
compile_dino=False, video=img)
|
| 24 |
+
dec = ViTDecoderConfig(latent_dim=32, bottleneck=StridedConvBottleneckConfig(stride=2),
|
| 25 |
+
vit_width=512, vit_depth=6, vit_num_heads=8, mlp_dim_multiplier=4,
|
| 26 |
+
qk_norm="layernorm", patch_size=16, patch_size_t=2, video=img,
|
| 27 |
+
activation_checkpointing=True)
|
| 28 |
+
cfg = VideoCodecConfig(encoder=enc, decoder=dec)
|
| 29 |
+
|
| 30 |
+
print("[consolidate] building codec (real HF DINOv3 backbone) + loading trained decoder...", flush=True)
|
| 31 |
+
model = VideoCodec(cfg, require_dino_weights=False).eval()
|
| 32 |
+
sd = torch.load(SRC, map_location="cpu", weights_only=False)
|
| 33 |
+
missing, unexpected = model.load_state_dict(sd["model"], strict=False)
|
| 34 |
+
print(f"[consolidate] loaded decoder (missing={len(missing)} keys are the frozen backbone; "
|
| 35 |
+
f"unexpected={len(unexpected)})", flush=True)
|
| 36 |
+
|
| 37 |
+
os.makedirs(OUT, exist_ok=True)
|
| 38 |
+
OmegaConf.save(OmegaConf.create({"model": {"architecture": {"config": cfg.model_dump()}}}),
|
| 39 |
+
os.path.join(OUT, VideoCodec.CONFIG_FILENAME))
|
| 40 |
+
ckpt_path = os.path.join(OUT, "codec.pth")
|
| 41 |
+
torch.save({"state_dict": model.state_dict()}, ckpt_path)
|
| 42 |
+
print(f"[consolidate] wrote {ckpt_path} + {VideoCodec.CONFIG_FILENAME}", flush=True)
|
| 43 |
+
|
| 44 |
+
# verify round-trip through MIRA's loader
|
| 45 |
+
loaded = VideoCodec.load_from_checkpoint(ckpt_path, device="cpu")
|
| 46 |
+
print(f"[ok] round-trips via load_from_checkpoint: "
|
| 47 |
+
f"{sum(p.numel() for p in loaded.parameters())/1e6:.0f}M params, "
|
| 48 |
+
f"latent_dim={loaded.latent_dim}, td={loaded.temporal_downsampling}, "
|
| 49 |
+
f"sd={loaded.spatial_downsampling}", flush=True)
|
scripts/eval_cs2_codec.py
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Check codec rendering quality: load a checkpoint, reconstruct a streamed CS2 clip, and save
|
| 2 |
+
ground-truth | reconstruction frames side by side (+ PSNR). Point at a checkpoint via CS2_CKPT,
|
| 3 |
+
else the newest in runs/cs2_codec/.
|
| 4 |
+
"""
|
| 5 |
+
import os, sys, glob, math
|
| 6 |
+
os.environ.setdefault("RS_DINO_HF", "facebook/dinov3-vitl16-pretrain-lvd1689m") # real backbone
|
| 7 |
+
sys.path.insert(0, "src")
|
| 8 |
+
import torch
|
| 9 |
+
from PIL import Image
|
| 10 |
+
|
| 11 |
+
from mira.codec import VideoCodec
|
| 12 |
+
from mira.codec.config import (
|
| 13 |
+
VideoCodecConfig, RAEEncoderConfig, ViTDecoderConfig, StridedConvBottleneckConfig, ImageConfig,
|
| 14 |
+
)
|
| 15 |
+
from mira.codec.viz import visualize_side_by_side
|
| 16 |
+
from mira.data.cs2_stream import create_cs2_loader
|
| 17 |
+
|
| 18 |
+
dev = "cuda" if torch.cuda.is_available() else "cpu"
|
| 19 |
+
T = int(os.environ.get("CS2_T", "24")); H, W, FPS = 288, 512, 24
|
| 20 |
+
|
| 21 |
+
# same architecture as training (weights, incl. frozen DINOv3, come from the checkpoint)
|
| 22 |
+
img = ImageConfig(height=H, width=W, channels=3, timesteps=T, fps=FPS)
|
| 23 |
+
enc = RAEEncoderConfig(latent_dim=32, rae_model="dinov3_vitl16",
|
| 24 |
+
aggregation_layers=[11, 13, 15, 17, 19, 21, 23],
|
| 25 |
+
bottleneck=StridedConvBottleneckConfig(stride=2, temporal_stride=2, noise_tau=0.0),
|
| 26 |
+
compile_dino=False, video=img)
|
| 27 |
+
dec = ViTDecoderConfig(latent_dim=32, bottleneck=StridedConvBottleneckConfig(stride=2),
|
| 28 |
+
vit_width=512, vit_depth=6, vit_num_heads=8, mlp_dim_multiplier=4,
|
| 29 |
+
qk_norm="layernorm", patch_size=16, patch_size_t=2, video=img,
|
| 30 |
+
activation_checkpointing=False)
|
| 31 |
+
model = VideoCodec(VideoCodecConfig(encoder=enc, decoder=dec), require_dino_weights=False).to(dev).eval()
|
| 32 |
+
|
| 33 |
+
ckpt = os.environ.get("CS2_CKPT") or max(glob.glob("runs/cs2_codec/*.pt"), key=os.path.getmtime)
|
| 34 |
+
sd = torch.load(ckpt, map_location=dev, weights_only=False)
|
| 35 |
+
model.load_state_dict(sd["model"], strict=False) # frozen backbone comes from HF, not the ckpt
|
| 36 |
+
print(f"[eval] loaded {ckpt} (step {sd.get('step')}, ema_loss {sd.get('ema_loss')})", flush=True)
|
| 37 |
+
|
| 38 |
+
loader = create_cs2_loader(subset=os.environ.get("CS2_SUBSET", "sample"), n_players=1, clip_len=T,
|
| 39 |
+
target_fps=FPS, frame_size=(H, W), batch_size=1, num_workers=0, infinite=True)
|
| 40 |
+
batch, meta = next(iter(loader))
|
| 41 |
+
with torch.no_grad():
|
| 42 |
+
out = model(batch.to(dev))
|
| 43 |
+
|
| 44 |
+
# PSNR over the whole clip (input/output are ~[-1,1] -> [0,1])
|
| 45 |
+
gt = (out.input_video * 0.5 + 0.5).clamp(0, 1)
|
| 46 |
+
pr = (out.output_video * 0.5 + 0.5).clamp(0, 1)
|
| 47 |
+
mse = torch.mean((gt - pr) ** 2).item()
|
| 48 |
+
psnr = 10 * math.log10(1.0 / mse) if mse > 0 else float("inf")
|
| 49 |
+
print(f"[eval] reconstruction PSNR = {psnr:.2f} dB (higher is better; ~20 rough, 28+ good)", flush=True)
|
| 50 |
+
|
| 51 |
+
viz = visualize_side_by_side(out)["viz_video"] # (B,T,C,H,2W) uint8, GT|recon
|
| 52 |
+
viz = viz[0] # (T,C,H,2W)
|
| 53 |
+
outdir = os.path.join(os.path.dirname(ckpt), "recon"); os.makedirs(outdir, exist_ok=True)
|
| 54 |
+
saved = []
|
| 55 |
+
for i in {0, T // 2, T - 1}:
|
| 56 |
+
im = viz[i].permute(1, 2, 0).cpu().numpy() # (H, 2W, C)
|
| 57 |
+
p = os.path.join(outdir, f"step{sd.get('step')}_frame{i:02d}_gt-left_recon-right.png")
|
| 58 |
+
Image.fromarray(im).save(p); saved.append(p)
|
| 59 |
+
print("[eval] saved (left = ground truth, right = reconstruction):", flush=True)
|
| 60 |
+
for p in saved:
|
| 61 |
+
print(" ", p, flush=True)
|
scripts/train_cs2_codec.py
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Minimal codec training on STREAMED CS2-10k — proves the streamed data -> MIRA model -> loss
|
| 2 |
+
pipeline end to end, in the existing venv (no pixi, no hydra). Pure L1 reconstruction so it needs
|
| 3 |
+
NO external weights; `require_dino_weights=False` builds the DINOv3 backbone with random weights
|
| 4 |
+
(swap in real gated DINOv3 later for a real RAE codec). Small config to fit 16GB and run fast.
|
| 5 |
+
"""
|
| 6 |
+
import os, sys, time, traceback
|
| 7 |
+
sys.path.insert(0, "src")
|
| 8 |
+
import torch
|
| 9 |
+
|
| 10 |
+
from mira.codec import VideoCodec, CodecLoss
|
| 11 |
+
from mira.codec.config import (
|
| 12 |
+
VideoCodecConfig, RAEEncoderConfig, ViTDecoderConfig,
|
| 13 |
+
StridedConvBottleneckConfig, ImageConfig,
|
| 14 |
+
)
|
| 15 |
+
from mira.codec.loss import CodecLossWeights
|
| 16 |
+
from mira.data.cs2_stream import create_cs2_loader
|
| 17 |
+
|
| 18 |
+
dev = "cuda" if torch.cuda.is_available() else "cpu"
|
| 19 |
+
# MIRA codec specs (paper Sec 6.3 / Table 9): per-view 288x512, 40-frame clips, DINOv3-L backbone,
|
| 20 |
+
# 1 latent per 32x32 px at half the input rate. CS2 is 48fps -> 24fps (48 isn't divisible by 20).
|
| 21 |
+
# Env knobs for 16GB tuning: CS2_T (frames, default 40), CS2_BS (batch, default 1), CS2_SUBSET.
|
| 22 |
+
T = int(os.environ.get("CS2_T", "40"))
|
| 23 |
+
FPS = 24
|
| 24 |
+
H, W = 288, 512
|
| 25 |
+
img = ImageConfig(height=H, width=W, channels=3, timesteps=T, fps=FPS)
|
| 26 |
+
enc = RAEEncoderConfig(
|
| 27 |
+
latent_dim=32, rae_model="dinov3_vitl16",
|
| 28 |
+
aggregation_layers=[11, 13, 15, 17, 19, 21, 23],
|
| 29 |
+
bottleneck=StridedConvBottleneckConfig(stride=2, temporal_stride=2, noise_tau=0.0),
|
| 30 |
+
compile_dino=False, video=img,
|
| 31 |
+
)
|
| 32 |
+
dec = ViTDecoderConfig(
|
| 33 |
+
latent_dim=32, bottleneck=StridedConvBottleneckConfig(stride=2),
|
| 34 |
+
# decoder size from env (defaults small/16GB; paper ViT-XL ~ 1152/28/16 on big GPUs)
|
| 35 |
+
vit_width=int(os.environ.get("CODEC_VIT_WIDTH", "512")),
|
| 36 |
+
vit_depth=int(os.environ.get("CODEC_VIT_DEPTH", "6")),
|
| 37 |
+
vit_num_heads=int(os.environ.get("CODEC_VIT_HEADS", "8")),
|
| 38 |
+
mlp_dim_multiplier=4,
|
| 39 |
+
qk_norm="layernorm", patch_size=16, patch_size_t=2, video=img, # 16*bottleneck_stride(2)=32 == encoder /32
|
| 40 |
+
activation_checkpointing=True,
|
| 41 |
+
)
|
| 42 |
+
cfg = VideoCodecConfig(encoder=enc, decoder=dec)
|
| 43 |
+
|
| 44 |
+
# Real gated DINOv3-L weights are used iff RS_DINO_WEIGHTS_DIR points at them (download from Meta:
|
| 45 |
+
# dinov3_vitl16_pretrain_lvd1689m-*.pth). Otherwise fall back to a random backbone so it still runs.
|
| 46 |
+
want_dino = bool(os.environ.get("RS_DINO_WEIGHTS_DIR"))
|
| 47 |
+
print(f"[cs2-codec] device={dev} T={T} {H}x{W}@{FPS}fps real_dino_weights={want_dino}", flush=True)
|
| 48 |
+
try:
|
| 49 |
+
model = VideoCodec(cfg, require_dino_weights=want_dino).to(dev).train()
|
| 50 |
+
except FileNotFoundError:
|
| 51 |
+
print("[cs2-codec] RS_DINO_WEIGHTS_DIR set but weights not found; using RANDOM backbone. "
|
| 52 |
+
"For a real codec, put dinov3_vitl16_pretrain_lvd1689m-*.pth in that dir.", flush=True)
|
| 53 |
+
model = VideoCodec(cfg, require_dino_weights=False).to(dev).train()
|
| 54 |
+
except Exception:
|
| 55 |
+
traceback.print_exc(); sys.exit(3)
|
| 56 |
+
|
| 57 |
+
print(f"[cs2-codec] params: {sum(p.numel() for p in model.parameters())/1e6:.1f}M", flush=True)
|
| 58 |
+
loss_fn = CodecLoss(CodecLossWeights(
|
| 59 |
+
loss_mae=1.0, loss_lpips_perceptual=0.0, loss_dino_latent_consistency=0.0, auto_weight=False,
|
| 60 |
+
)).to(dev)
|
| 61 |
+
opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
|
| 62 |
+
|
| 63 |
+
loader = create_cs2_loader(
|
| 64 |
+
subset=os.environ.get("CS2_SUBSET", "sample"), n_players=1, clip_len=T, target_fps=FPS,
|
| 65 |
+
frame_size=(H, W), batch_size=int(os.environ.get("CS2_BS", "1")),
|
| 66 |
+
num_workers=int(os.environ.get("CS2_WORKERS", "0")), # >0 on Linux (Modal) to parallelize decode
|
| 67 |
+
infinite=True,
|
| 68 |
+
)
|
| 69 |
+
print("[cs2-codec] streaming CS2 sample split; first batch downloads a few clips...", flush=True)
|
| 70 |
+
|
| 71 |
+
from pathlib import Path
|
| 72 |
+
STEPS = int(os.environ.get("CS2_STEPS", "20000"))
|
| 73 |
+
CKPT_EVERY = int(os.environ.get("CS2_CKPT_EVERY", "500"))
|
| 74 |
+
ckpt_dir = Path(os.environ.get("CS2_OUT", "runs/cs2_codec"))
|
| 75 |
+
ckpt_dir.mkdir(parents=True, exist_ok=True)
|
| 76 |
+
print(f"[cs2-codec] REAL RUN: {STEPS} steps, checkpoint every {CKPT_EVERY} -> {ckpt_dir}", flush=True)
|
| 77 |
+
|
| 78 |
+
# Resume from the newest checkpoint (or CS2_RESUME) so restarts don't lose progress.
|
| 79 |
+
import glob
|
| 80 |
+
_start_step, _ema0 = 0, None
|
| 81 |
+
_resume = os.environ.get("CS2_RESUME") or (sorted(glob.glob(str(ckpt_dir / "codec_0*.pt"))) or [None])[-1]
|
| 82 |
+
def _trainable_sd():
|
| 83 |
+
# exclude the frozen DINOv3 backbone (reloaded from HF/hub) -> small, portable checkpoints
|
| 84 |
+
return {k: v for k, v in model.state_dict().items() if "rae_dino.dino_model" not in k}
|
| 85 |
+
|
| 86 |
+
if _resume and os.path.exists(_resume):
|
| 87 |
+
_sd = torch.load(_resume, map_location=dev, weights_only=False)
|
| 88 |
+
model.load_state_dict(_sd["model"], strict=False) # backbone already loaded (HF/hub)
|
| 89 |
+
_start_step = int(_sd.get("step", 0)); _ema0 = _sd.get("ema_loss")
|
| 90 |
+
print(f"[cs2-codec] resumed from {_resume} at step {_start_step} (ema_loss={_ema0})", flush=True)
|
| 91 |
+
|
| 92 |
+
# M.2 thermal duty-cycle: every CS2_PAUSE_EVERY steps, sleep CS2_PAUSE_SECS to let the SSD cool.
|
| 93 |
+
PAUSE_EVERY = int(os.environ.get("CS2_PAUSE_EVERY", "1000"))
|
| 94 |
+
PAUSE_SECS = int(os.environ.get("CS2_PAUSE_SECS", "900"))
|
| 95 |
+
|
| 96 |
+
step, ema = _start_step, _ema0
|
| 97 |
+
_pace_t0, _pace_s0 = time.time(), _start_step
|
| 98 |
+
for batch, meta in loader:
|
| 99 |
+
batch = batch.to(dev)
|
| 100 |
+
out = model(batch)
|
| 101 |
+
losses = loss_fn(out, global_step=step)
|
| 102 |
+
total = losses["loss"] if "loss" in losses else sum(v for v in losses.values() if v.ndim == 0)
|
| 103 |
+
opt.zero_grad(); total.backward(); opt.step()
|
| 104 |
+
ema = float(total) if ema is None else 0.98 * ema + 0.02 * float(total)
|
| 105 |
+
if step % 20 == 0:
|
| 106 |
+
print(f"[cs2-codec] step {step:6d} loss={float(total):.4f} ema={ema:.4f} "
|
| 107 |
+
f"({(time.time()-_pace_t0)/(step-_pace_s0+1):.2f}s/step)", flush=True)
|
| 108 |
+
step += 1
|
| 109 |
+
if step % CKPT_EVERY == 0:
|
| 110 |
+
torch.save({"step": step, "model": _trainable_sd(), "config": cfg.model_dump(),
|
| 111 |
+
"ema_loss": ema}, ckpt_dir / f"codec_{step:06d}.pt")
|
| 112 |
+
print(f"[cs2-codec] saved {ckpt_dir / f'codec_{step:06d}.pt'} (ema_loss={ema:.4f})", flush=True)
|
| 113 |
+
keep = int(os.environ.get("CS2_KEEP", "2")) # cap disk: keep only the newest N
|
| 114 |
+
for old in sorted(ckpt_dir.glob("codec_0*.pt"))[:-keep]:
|
| 115 |
+
old.unlink(missing_ok=True)
|
| 116 |
+
if step >= STEPS:
|
| 117 |
+
break
|
| 118 |
+
if PAUSE_EVERY and step % PAUSE_EVERY == 0: # checkpoint just saved (1000%500==0)
|
| 119 |
+
import datetime as _dt
|
| 120 |
+
print(f"[cs2-codec] PAUSE {PAUSE_SECS}s at step {step} for M.2 cooldown "
|
| 121 |
+
f"(resume ~{(_dt.datetime.now()+_dt.timedelta(seconds=PAUSE_SECS)).strftime('%H:%M:%S')})",
|
| 122 |
+
flush=True)
|
| 123 |
+
time.sleep(PAUSE_SECS)
|
| 124 |
+
_pace_t0, _pace_s0 = time.time(), step # exclude the sleep from the pace estimate
|
| 125 |
+
print(f"[cs2-codec] RESUME at step {step}", flush=True)
|
| 126 |
+
torch.save({"step": step, "model": _trainable_sd(), "config": cfg.model_dump(), "ema_loss": ema},
|
| 127 |
+
ckpt_dir / "codec_final.pt")
|
| 128 |
+
print(f"[cs2-codec] DONE {step} steps, final ema_loss={ema:.4f} -> {ckpt_dir/'codec_final.pt'}", flush=True)
|
scripts/train_cs2_wm.py
ADDED
|
@@ -0,0 +1,121 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Train a small MIRA latent world model on the CS2 codec, and test action-controllability.
|
| 2 |
+
|
| 3 |
+
train: python scripts/train_cs2_wm.py
|
| 4 |
+
test : python scripts/train_cs2_wm.py --test (seed real context, feed fixed actions, generate)
|
| 5 |
+
|
| 6 |
+
Frozen codec is loaded from the consolidated checkpoint; a tiny flow-matching diffusion transformer
|
| 7 |
+
learns to predict the next codec latent conditioned on CS2 keyboard+mouse actions.
|
| 8 |
+
"""
|
| 9 |
+
import os, sys, time, argparse
|
| 10 |
+
os.environ.setdefault("RS_DINO_HF", "facebook/dinov3-vitl16-pretrain-lvd1689m")
|
| 11 |
+
sys.path.insert(0, "src")
|
| 12 |
+
import torch
|
| 13 |
+
|
| 14 |
+
from mira.world_model.config import LatentWorldModelConfig
|
| 15 |
+
from mira.world_model.latent_world_model import LatentWorldModel
|
| 16 |
+
from mira.world_model.actions_config import ActionConfig
|
| 17 |
+
from mira.ml.image_config import ImageConfig
|
| 18 |
+
from mira.data.cs2_stream import create_cs2_loader, CS2_KEYS
|
| 19 |
+
|
| 20 |
+
dev = "cuda" if torch.cuda.is_available() else "cpu"
|
| 21 |
+
T = int(os.environ.get("CS2_T", "16"))
|
| 22 |
+
CODEC = os.environ.get("CS2_CODEC", "runs/cs2_codec_consolidated/codec.pth")
|
| 23 |
+
OUT = os.environ.get("CS2_WM_OUT", "runs/cs2_wm")
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def build_model():
|
| 27 |
+
actions = ActionConfig(valid_keys=list(CS2_KEYS), source_fps=24, target_fps=24)
|
| 28 |
+
video = ImageConfig(height=288, width=512, channels=3, timesteps=T, fps=24)
|
| 29 |
+
cfg = LatentWorldModelConfig(
|
| 30 |
+
actions=actions, video=video, codec_checkpoint=CODEC,
|
| 31 |
+
latent_mean_std=[0.0, 1.0], # identity norm (we didn't estimate latent stats)
|
| 32 |
+
causal=True, use_clean_past=True, learned_temporal_pool=True,
|
| 33 |
+
use_codec_posterior_mean=True, attention_gating=True, ada_attn_ln=True,
|
| 34 |
+
patch_size=1, n_register_tokens=0, n_context_frames=max(2, T // 2 - 2),
|
| 35 |
+
dropout_action_prob=0.1,
|
| 36 |
+
# size from env (defaults = tiny/16GB-safe; scale up on big GPUs, e.g. 1B: 2048/16/4/16)
|
| 37 |
+
hidden_dim=int(os.environ.get("WM_HIDDEN", "384")),
|
| 38 |
+
n_head=int(os.environ.get("WM_HEADS", "6")),
|
| 39 |
+
n_kv_head=int(os.environ.get("WM_KV", "2")),
|
| 40 |
+
n_layers=int(os.environ.get("WM_LAYERS", "4")),
|
| 41 |
+
time_attention_every=int(os.environ.get("WM_TATT", "2")),
|
| 42 |
+
activation_checkpointing=os.environ.get("WM_ACT_CKPT", "0") == "1",
|
| 43 |
+
)
|
| 44 |
+
return LatentWorldModel(cfg).to(dev)
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def train():
|
| 48 |
+
from pathlib import Path
|
| 49 |
+
model = build_model().train()
|
| 50 |
+
n_tr = sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6
|
| 51 |
+
print(f"[wm] trainable params: {n_tr:.1f}M (+ frozen codec)", flush=True)
|
| 52 |
+
opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
|
| 53 |
+
loader = create_cs2_loader(subset=os.environ.get("CS2_SUBSET", "sample"), n_players=1, clip_len=T,
|
| 54 |
+
target_fps=24, frame_size=(288, 512),
|
| 55 |
+
batch_size=int(os.environ.get("WM_BATCH", "1")),
|
| 56 |
+
num_workers=int(os.environ.get("CS2_WORKERS", "0")), infinite=True)
|
| 57 |
+
ckpt_dir = Path(OUT); ckpt_dir.mkdir(parents=True, exist_ok=True)
|
| 58 |
+
STEPS = int(os.environ.get("CS2_WM_STEPS", "5000"))
|
| 59 |
+
step, ema, t0 = 0, None, time.time()
|
| 60 |
+
for batch, meta in loader:
|
| 61 |
+
losses = model(batch.to(dev))
|
| 62 |
+
total = losses["loss_total"]
|
| 63 |
+
opt.zero_grad(); total.backward(); opt.step()
|
| 64 |
+
ema = float(total) if ema is None else 0.98 * ema + 0.02 * float(total)
|
| 65 |
+
if step % 20 == 0:
|
| 66 |
+
print(f"[wm] step {step:5d} loss={float(total):.4f} ema={ema:.4f} "
|
| 67 |
+
f"({(time.time()-t0)/(step+1):.2f}s/step)", flush=True)
|
| 68 |
+
step += 1
|
| 69 |
+
if step % 500 == 0:
|
| 70 |
+
# exclude the frozen codec (rebuilt from CS2_CODEC) -> ~50MB instead of ~1.3GB
|
| 71 |
+
wm_only = {k: v for k, v in model.state_dict().items() if not k.startswith("codec.")}
|
| 72 |
+
torch.save({"step": step, "model": wm_only, "ema": ema}, ckpt_dir / f"wm_{step:06d}.pt")
|
| 73 |
+
for old in sorted(ckpt_dir.glob("wm_0*.pt"))[:-2]:
|
| 74 |
+
old.unlink(missing_ok=True)
|
| 75 |
+
print(f"[wm] saved wm_{step:06d}.pt (ema={ema:.4f})", flush=True)
|
| 76 |
+
if step >= STEPS:
|
| 77 |
+
break
|
| 78 |
+
print("[wm] done", flush=True)
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
@torch.no_grad()
|
| 82 |
+
def test():
|
| 83 |
+
"""Seed a real context clip, then generate under three fixed action regimes and save frames."""
|
| 84 |
+
import glob, numpy as np
|
| 85 |
+
from PIL import Image
|
| 86 |
+
from mira.world_model.latent_world_model import WorldModelInferenceConfig
|
| 87 |
+
model = build_model().eval()
|
| 88 |
+
ck = os.environ.get("CS2_WM_CKPT") or (sorted(glob.glob(f"{OUT}/wm_0*.pt")) or [None])[-1]
|
| 89 |
+
if ck:
|
| 90 |
+
model.load_state_dict(torch.load(ck, map_location=dev, weights_only=False)["model"], strict=False)
|
| 91 |
+
print(f"[wm-test] loaded {ck}", flush=True)
|
| 92 |
+
loader = create_cs2_loader(subset="sample", n_players=1, clip_len=T, target_fps=24,
|
| 93 |
+
frame_size=(288, 512), batch_size=1, num_workers=0, infinite=True)
|
| 94 |
+
batch, _ = next(iter(loader))
|
| 95 |
+
|
| 96 |
+
# key indices in the CS2 vocab
|
| 97 |
+
kW, kMOUSE = CS2_KEYS.index("W"), None
|
| 98 |
+
regimes = {
|
| 99 |
+
"idle": lambda a: (a.key_presses.zero_(), a.mouse_movements.zero_()),
|
| 100 |
+
"forward": lambda a: (a.key_presses.zero_().__setitem__((slice(None), slice(None), kW), 1),
|
| 101 |
+
a.mouse_movements.zero_()),
|
| 102 |
+
"turn_right": lambda a: (a.key_presses.zero_(),
|
| 103 |
+
a.mouse_movements.zero_().__setitem__((slice(None), slice(None), 0), 25.0)),
|
| 104 |
+
}
|
| 105 |
+
icfg = WorldModelInferenceConfig() if False else None
|
| 106 |
+
outdir = os.path.join(OUT, "control"); os.makedirs(outdir, exist_ok=True)
|
| 107 |
+
for name, override in regimes.items():
|
| 108 |
+
b = batch.clone()
|
| 109 |
+
override(b.actions)
|
| 110 |
+
out = model.inference(b.to(dev), progress_bar=False)
|
| 111 |
+
vid = out.output_video[0] # (T,C,H,W) in [-1,1]
|
| 112 |
+
img = ((vid[-1] * 0.5 + 0.5).clamp(0, 1) * 255).byte().permute(1, 2, 0).cpu().numpy()
|
| 113 |
+
p = os.path.join(outdir, f"gen_{name}_lastframe.png")
|
| 114 |
+
Image.fromarray(img).save(p)
|
| 115 |
+
print(f"[wm-test] {name}: saved {p}", flush=True)
|
| 116 |
+
print("[wm-test] compare the three frames: forward/turn should differ from idle if controllable.", flush=True)
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
if __name__ == "__main__":
|
| 120 |
+
ap = argparse.ArgumentParser(); ap.add_argument("--test", action="store_true"); a = ap.parse_args()
|
| 121 |
+
(test if a.test else train)()
|
src/mira/data/cs2_stream.py
ADDED
|
@@ -0,0 +1,353 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Streaming CS2-10k dataset for MIRA (no local dataset build).
|
| 2 |
+
|
| 3 |
+
RekaAI/CS2-10k is a WebDataset of first-person CS2 rounds. `index.parquet` lists every clip with:
|
| 4 |
+
match_id, video_path, parquet_path, map, player_index, round_number, team, fps(48), total_time,
|
| 5 |
+
width, height, match_num_clips, match_num_rounds, match_num_players, shard
|
| 6 |
+
Grouping key for MIRA's multi-perspective samples is (match_id, round_number): fixing both and
|
| 7 |
+
varying player_index gives up to `match_num_players` (=10) synchronized first-person views of the
|
| 8 |
+
same round — the CS2 analogue of MIRA's 4 Rocket League views.
|
| 9 |
+
|
| 10 |
+
This module streams those groups straight from the Hub and yields MIRA's own
|
| 11 |
+
`(VideoActionBatch, list[ClipMeta])` batches, so it drops into MIRA's trainer in place of
|
| 12 |
+
`create_loader` with zero on-disk dataset. It reuses MIRA's `decode_frames`, `KeyVocab`,
|
| 13 |
+
`ActionTensors`, `ClipMeta`, `VideoActionBatch`, and `_collate`.
|
| 14 |
+
|
| 15 |
+
Two important, honest caveats:
|
| 16 |
+
* ALIGNMENT: CS2 gives no shared per-frame tick. Each round's per-player clips are 48fps and
|
| 17 |
+
(empirically) start at round start, so we align at frame 0 and truncate a group to the shortest
|
| 18 |
+
clip. Refine later with position/rotation cross-checks if needed.
|
| 19 |
+
* SCALE / STREAMING UNIT: the 63TB main split is tar-sharded and a round's POVs are scattered
|
| 20 |
+
across many shards, so complete groups are NOT co-located in one tar. This loader therefore
|
| 21 |
+
fetches clips as INDIVIDUAL files by path (works today for the untarred `sample/` split — 3 full
|
| 22 |
+
matches — and for any individual-file mirror). Training on the full tarred split at scale needs
|
| 23 |
+
either re-sharding by (match,round) or a per-group multi-shard fetch; that's a separate step.
|
| 24 |
+
"""
|
| 25 |
+
|
| 26 |
+
from __future__ import annotations
|
| 27 |
+
|
| 28 |
+
import io
|
| 29 |
+
import random
|
| 30 |
+
from collections import defaultdict
|
| 31 |
+
from itertools import count
|
| 32 |
+
from typing import Any, Iterator
|
| 33 |
+
|
| 34 |
+
import numpy as np
|
| 35 |
+
import torch
|
| 36 |
+
from torch.utils.data import DataLoader, IterableDataset, get_worker_info
|
| 37 |
+
|
| 38 |
+
from huggingface_hub import hf_hub_download
|
| 39 |
+
import pandas as pd
|
| 40 |
+
|
| 41 |
+
import av # ffmpeg-backed decode (works in this venv; avoids the torchcodec dependency)
|
| 42 |
+
|
| 43 |
+
from .actions import KeyVocab
|
| 44 |
+
from .batch import VideoActionBatch
|
| 45 |
+
from .clips import compute_stride
|
| 46 |
+
from .training_loader import ClipMeta, _collate # reuse MIRA's collate + meta type
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
def _decode_av(mp4_bytes: bytes, frame_indices: list[int], frame_size: tuple[int, int] | None):
|
| 50 |
+
"""Decode the given source-frame indices from mp4 bytes to (T, C, H, W) uint8, via PyAV."""
|
| 51 |
+
want = set(frame_indices)
|
| 52 |
+
hi = max(frame_indices)
|
| 53 |
+
grabbed: dict[int, np.ndarray] = {}
|
| 54 |
+
with av.open(io.BytesIO(mp4_bytes)) as container:
|
| 55 |
+
for i, frame in enumerate(container.decode(container.streams.video[0])):
|
| 56 |
+
if i in want:
|
| 57 |
+
grabbed[i] = frame.to_ndarray(format="rgb24") # (H, W, 3) uint8
|
| 58 |
+
if i >= hi:
|
| 59 |
+
break
|
| 60 |
+
arr = np.stack([grabbed[i] for i in frame_indices]) # (T, H, W, 3)
|
| 61 |
+
t = torch.from_numpy(arr).permute(0, 3, 1, 2).contiguous() # (T, C, H, W) uint8
|
| 62 |
+
if frame_size is not None:
|
| 63 |
+
t = torch.nn.functional.interpolate(
|
| 64 |
+
t.float(), size=frame_size, mode="bilinear", align_corners=False
|
| 65 |
+
).clamp(0, 255).to(torch.uint8)
|
| 66 |
+
return t
|
| 67 |
+
|
| 68 |
+
# CS2 held-key characters (per-frame `actions` string), stable multi-hot order. '-' == no input.
|
| 69 |
+
CS2_KEYS: tuple[str, ...] = ("W", "A", "S", "D", "J", "C", "R", "V", "[", "]")
|
| 70 |
+
|
| 71 |
+
REPO_ID = "RekaAI/CS2-10k"
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def _rank_world() -> tuple[int, int]:
|
| 75 |
+
if torch.distributed.is_available() and torch.distributed.is_initialized():
|
| 76 |
+
return torch.distributed.get_rank(), torch.distributed.get_world_size()
|
| 77 |
+
return 0, 1
|
| 78 |
+
|
| 79 |
+
|
| 80 |
+
def _fetch_bytes(path: str) -> bytes:
|
| 81 |
+
"""Fetch an INDIVIDUAL file by repo path (works for the untarred `sample/` split)."""
|
| 82 |
+
local = hf_hub_download(REPO_ID, path, repo_type="dataset")
|
| 83 |
+
with open(local, "rb") as f:
|
| 84 |
+
return f.read()
|
| 85 |
+
|
| 86 |
+
|
| 87 |
+
# --- ranged tar-member fetch for the FULL (tarred) split ------------------------------------------
|
| 88 |
+
# A round's POVs are scattered across many ~2GB tars. We open each tar over HfFileSystem (a seekable,
|
| 89 |
+
# HTTP-range-backed file) with tarfile mode="r", which walks the member headers via small ranged
|
| 90 |
+
# reads ONCE per shard, then extracts a single member with one ranged read of just its bytes — so we
|
| 91 |
+
# never download a whole 2GB tar. Open TarFiles are cached per shard.
|
| 92 |
+
import tarfile
|
| 93 |
+
|
| 94 |
+
_HF_FS = None
|
| 95 |
+
_TAR_CACHE: dict[str, tuple] = {}
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def _hf_fs():
|
| 99 |
+
global _HF_FS
|
| 100 |
+
if _HF_FS is None:
|
| 101 |
+
from huggingface_hub import HfFileSystem
|
| 102 |
+
_HF_FS = HfFileSystem()
|
| 103 |
+
return _HF_FS
|
| 104 |
+
|
| 105 |
+
|
| 106 |
+
def _open_shard(shard: str):
|
| 107 |
+
if shard not in _TAR_CACHE:
|
| 108 |
+
f = _hf_fs().open(f"datasets/{REPO_ID}/{shard}", "rb") # seekable, range-backed
|
| 109 |
+
tf = tarfile.open(fileobj=f, mode="r") # header walk (ranged) once
|
| 110 |
+
members = {m.name.rsplit("/", 1)[-1]: m for m in tf.getmembers()}
|
| 111 |
+
if len(_TAR_CACHE) > 24: # bound open handles
|
| 112 |
+
old = next(iter(_TAR_CACHE))
|
| 113 |
+
try:
|
| 114 |
+
_TAR_CACHE.pop(old)[0].close()
|
| 115 |
+
except Exception:
|
| 116 |
+
pass
|
| 117 |
+
_TAR_CACHE[shard] = (tf, members)
|
| 118 |
+
return _TAR_CACHE[shard]
|
| 119 |
+
|
| 120 |
+
|
| 121 |
+
def _fetch_from_tar(shard: str, repo_path: str) -> bytes:
|
| 122 |
+
tf, members = _open_shard(shard)
|
| 123 |
+
m = members[repo_path.rsplit("/", 1)[-1]]
|
| 124 |
+
return tf.extractfile(m).read()
|
| 125 |
+
|
| 126 |
+
|
| 127 |
+
def _build_action_arrays(
|
| 128 |
+
frame_data: np.ndarray, vocab: KeyVocab, stride: int
|
| 129 |
+
) -> tuple[torch.Tensor, torch.Tensor]:
|
| 130 |
+
"""From a clip's 48fps per-frame records build downsampled (n_steps, n_keys) int32 multi-hot key
|
| 131 |
+
presses (OR-ed over each stride window) and (n_steps, 2) float32 mean mouse deltas."""
|
| 132 |
+
n_keys = len(vocab)
|
| 133 |
+
n_steps = len(frame_data) // stride
|
| 134 |
+
keys = torch.zeros((n_steps, n_keys), dtype=torch.int32)
|
| 135 |
+
mouse = torch.zeros((n_steps, 2), dtype=torch.float32)
|
| 136 |
+
for s in range(n_steps):
|
| 137 |
+
mx = my = 0.0
|
| 138 |
+
for j in range(s * stride, (s + 1) * stride):
|
| 139 |
+
rec = frame_data[j]
|
| 140 |
+
a = rec["actions"]
|
| 141 |
+
if a and a != "-":
|
| 142 |
+
for ch in a:
|
| 143 |
+
idx = vocab._index.get(ch)
|
| 144 |
+
if idx is not None:
|
| 145 |
+
keys[s, idx] = 1
|
| 146 |
+
mx += float(rec["mouse_x_delta"])
|
| 147 |
+
my += float(rec["mouse_y_delta"])
|
| 148 |
+
mouse[s, 0] = mx / stride
|
| 149 |
+
mouse[s, 1] = my / stride
|
| 150 |
+
return keys, mouse
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
class Cs2StreamingDataset(IterableDataset):
|
| 154 |
+
"""Streams (match_id, round_number) POV groups from CS2-10k as MIRA per-perspective samples.
|
| 155 |
+
|
| 156 |
+
Yields dicts shaped exactly like MIRA's training_loader `_decode_sample` output
|
| 157 |
+
({"video","actions","metadata"}), so `_collate` turns a batch into `(VideoActionBatch, [ClipMeta])`.
|
| 158 |
+
"""
|
| 159 |
+
|
| 160 |
+
def __init__(
|
| 161 |
+
self,
|
| 162 |
+
index: pd.DataFrame,
|
| 163 |
+
*,
|
| 164 |
+
vocab: KeyVocab,
|
| 165 |
+
clip_len: int = 16,
|
| 166 |
+
target_fps: int = 16,
|
| 167 |
+
source_fps: int = 48,
|
| 168 |
+
n_players: int = 4,
|
| 169 |
+
frame_size: tuple[int, int] | None = None,
|
| 170 |
+
mouse_sensitivity: float = 1.0,
|
| 171 |
+
shuffle: bool = True,
|
| 172 |
+
infinite: bool = True,
|
| 173 |
+
seed: int = 2025,
|
| 174 |
+
from_tar: bool = False,
|
| 175 |
+
) -> None:
|
| 176 |
+
super().__init__()
|
| 177 |
+
self.index = index
|
| 178 |
+
self.from_tar = from_tar
|
| 179 |
+
self.vocab = vocab
|
| 180 |
+
self.clip_len = clip_len
|
| 181 |
+
self.target_fps = target_fps
|
| 182 |
+
self.source_fps = source_fps
|
| 183 |
+
self.n_players = n_players
|
| 184 |
+
self.frame_size = frame_size
|
| 185 |
+
self.mouse_sensitivity = mouse_sensitivity
|
| 186 |
+
self.shuffle = shuffle
|
| 187 |
+
self.infinite = infinite
|
| 188 |
+
self.seed = seed
|
| 189 |
+
self.stride = compute_stride(source_fps, target_fps)
|
| 190 |
+
# group rows into (match_id, round_number) -> [row, ...] sorted by player_index
|
| 191 |
+
groups: dict[tuple[str, int], list[dict]] = defaultdict(list)
|
| 192 |
+
for r in index.to_dict("records"):
|
| 193 |
+
groups[(r["match_id"], int(r["round_number"]))].append(r)
|
| 194 |
+
# keep only groups with enough perspectives to form an n_players block
|
| 195 |
+
self.groups: list[list[dict]] = [
|
| 196 |
+
sorted(v, key=lambda r: r["player_index"])
|
| 197 |
+
for v in groups.values()
|
| 198 |
+
if len(v) >= n_players
|
| 199 |
+
]
|
| 200 |
+
|
| 201 |
+
def _my_groups(self) -> list[list[dict]]:
|
| 202 |
+
rank, world = _rank_world()
|
| 203 |
+
info = get_worker_info()
|
| 204 |
+
wid, nw = (info.id, info.num_workers) if info else (0, 1)
|
| 205 |
+
return self.groups[rank::world][wid::nw]
|
| 206 |
+
|
| 207 |
+
def _load_group(self, rows: list[dict]) -> tuple[list[bytes], list[np.ndarray], list[dict]]:
|
| 208 |
+
import io as _io
|
| 209 |
+
vids, fdata, meta = [], [], []
|
| 210 |
+
for r in rows[: self.n_players]:
|
| 211 |
+
if self.from_tar: # full split: ranged tar members
|
| 212 |
+
vids.append(_fetch_from_tar(r["shard"], r["video_path"]))
|
| 213 |
+
pq = pd.read_parquet(_io.BytesIO(_fetch_from_tar(r["shard"], r["parquet_path"])))
|
| 214 |
+
else: # sample split: individual files
|
| 215 |
+
vids.append(_fetch_bytes(r["video_path"]))
|
| 216 |
+
pq = pd.read_parquet(hf_hub_download(REPO_ID, r["parquet_path"], repo_type="dataset"))
|
| 217 |
+
fdata.append(pq.iloc[0]["frame_data"])
|
| 218 |
+
meta.append(r)
|
| 219 |
+
return vids, fdata, meta
|
| 220 |
+
|
| 221 |
+
def _samples_from_group(self, rows: list[dict]) -> Iterator[dict[str, Any]]:
|
| 222 |
+
vids, fdata, meta = self._load_group(rows)
|
| 223 |
+
n_src = min(len(fd) for fd in fdata) # align at frame 0, truncate to shortest
|
| 224 |
+
n_steps = n_src // self.stride
|
| 225 |
+
if n_steps < self.clip_len:
|
| 226 |
+
return
|
| 227 |
+
# per-perspective full-round action arrays, then slice per clip window
|
| 228 |
+
key_arrs, mouse_arrs = [], []
|
| 229 |
+
for fd in fdata:
|
| 230 |
+
k, m = _build_action_arrays(fd[: n_steps * self.stride], self.vocab, self.stride)
|
| 231 |
+
key_arrs.append(k)
|
| 232 |
+
mouse_arrs.append(m)
|
| 233 |
+
# Decode every target-fps frame of the clip ONCE per perspective, then slice windows from it.
|
| 234 |
+
# (Decoding per window re-decoded from the start of the mp4 each time -> O(n^2) and starved
|
| 235 |
+
# the GPU; this is a single linear pass.)
|
| 236 |
+
all_idx = [s * self.stride for s in range(n_steps)]
|
| 237 |
+
full = [_decode_av(vids[p], all_idx, self.frame_size) for p in range(self.n_players)]
|
| 238 |
+
clip_id = 0
|
| 239 |
+
for start in range(0, n_steps - self.clip_len + 1, self.clip_len):
|
| 240 |
+
frame_indices = all_idx[start : start + self.clip_len]
|
| 241 |
+
for p in range(self.n_players):
|
| 242 |
+
video = full[p][start : start + self.clip_len] # (T,C,H,W) uint8, already decoded
|
| 243 |
+
act = _make_action_tensors(
|
| 244 |
+
key_arrs[p][start : start + self.clip_len],
|
| 245 |
+
mouse_arrs[p][start : start + self.clip_len],
|
| 246 |
+
self.vocab,
|
| 247 |
+
self.mouse_sensitivity,
|
| 248 |
+
)
|
| 249 |
+
yield {
|
| 250 |
+
"video": video,
|
| 251 |
+
"actions": act,
|
| 252 |
+
"metadata": ClipMeta(
|
| 253 |
+
match_id=meta[p]["match_id"],
|
| 254 |
+
perspective=p,
|
| 255 |
+
player_id=int(meta[p]["player_index"]),
|
| 256 |
+
clip_id=clip_id,
|
| 257 |
+
chunk_idx=int(meta[p]["round_number"]),
|
| 258 |
+
frame_indices=list(frame_indices),
|
| 259 |
+
),
|
| 260 |
+
}
|
| 261 |
+
clip_id += 1
|
| 262 |
+
|
| 263 |
+
def __iter__(self) -> Iterator[dict[str, Any]]:
|
| 264 |
+
rank, _ = _rank_world()
|
| 265 |
+
info = get_worker_info()
|
| 266 |
+
rng = random.Random(self.seed + rank * 1024 + (info.id if info else 0))
|
| 267 |
+
for _epoch in count() if self.infinite else range(1):
|
| 268 |
+
order = self._my_groups()
|
| 269 |
+
if self.shuffle:
|
| 270 |
+
rng.shuffle(order)
|
| 271 |
+
for rows in order:
|
| 272 |
+
try:
|
| 273 |
+
yield from self._samples_from_group(rows)
|
| 274 |
+
except Exception as e: # a bad clip shouldn't kill the epoch
|
| 275 |
+
print(f"[cs2_stream] skipping group {rows[0]['match_id']}: {e}")
|
| 276 |
+
|
| 277 |
+
|
| 278 |
+
def _make_action_tensors(keys: torch.Tensor, mouse: torch.Tensor, vocab: KeyVocab, sens: float):
|
| 279 |
+
"""Build a MIRA ActionTensors (batch=1) from a clip window's key/mouse tensors."""
|
| 280 |
+
from mira.world_model.actions_config import ActionConfig, ActionTensors
|
| 281 |
+
|
| 282 |
+
cfg = ActionConfig(valid_keys=list(vocab.keys))
|
| 283 |
+
at = ActionTensors(config=cfg, batch_size=1)
|
| 284 |
+
at.key_presses = keys.unsqueeze(0).to(torch.int32) # (1, T, n_keys)
|
| 285 |
+
at.mouse_movements = mouse.unsqueeze(0).to(torch.float32) # (1, T, 2)
|
| 286 |
+
at.game_mouse_sensitivity = torch.full((1,), float(sens), dtype=torch.float32)
|
| 287 |
+
return at
|
| 288 |
+
|
| 289 |
+
|
| 290 |
+
def load_cs2_index(subset: str = "sample") -> pd.DataFrame:
|
| 291 |
+
"""Return the CS2 index as a DataFrame. `subset='sample'` uses the untarred 3-match `sample/`
|
| 292 |
+
split whose clips are individually fetchable (recommended for streaming today); `subset='full'`
|
| 293 |
+
uses the root index (tarred; see the module docstring caveat before using at scale)."""
|
| 294 |
+
if subset == "sample":
|
| 295 |
+
p = hf_hub_download(REPO_ID, "sample/index.parquet", repo_type="dataset")
|
| 296 |
+
df = pd.read_parquet(p)
|
| 297 |
+
# sample/ paths are relative to sample/; make them repo-relative
|
| 298 |
+
for col in ("video_path", "parquet_path"):
|
| 299 |
+
if col in df.columns:
|
| 300 |
+
df[col] = df[col].apply(lambda x: x if x.startswith("sample/") else f"sample/{x}")
|
| 301 |
+
return df
|
| 302 |
+
p = hf_hub_download(REPO_ID, "index.parquet", repo_type="dataset")
|
| 303 |
+
return pd.read_parquet(p)
|
| 304 |
+
|
| 305 |
+
|
| 306 |
+
def create_cs2_loader(
|
| 307 |
+
*,
|
| 308 |
+
subset: str = "sample",
|
| 309 |
+
index: pd.DataFrame | None = None,
|
| 310 |
+
maps: list[str] | None = None,
|
| 311 |
+
clip_len: int = 16,
|
| 312 |
+
target_fps: int = 16,
|
| 313 |
+
n_players: int = 4,
|
| 314 |
+
batch_size: int = 4,
|
| 315 |
+
num_workers: int = 0,
|
| 316 |
+
frame_size: tuple[int, int] | None = None,
|
| 317 |
+
valid_keys: list[str] | None = None,
|
| 318 |
+
shuffle: bool = True,
|
| 319 |
+
infinite: bool = True,
|
| 320 |
+
seed: int = 2025,
|
| 321 |
+
prefetch_factor: int = 2,
|
| 322 |
+
) -> DataLoader:
|
| 323 |
+
"""Drop-in replacement for MIRA's `create_loader`, streaming CS2-10k from the Hub (no disk).
|
| 324 |
+
|
| 325 |
+
Yields `(VideoActionBatch, list[ClipMeta])`. `n_players` perspectives per (match, round) are
|
| 326 |
+
grouped contiguously so the collate stacks them into one multi-perspective block, exactly as
|
| 327 |
+
MIRA's own loader does.
|
| 328 |
+
"""
|
| 329 |
+
if index is None:
|
| 330 |
+
index = load_cs2_index(subset)
|
| 331 |
+
if maps:
|
| 332 |
+
index = index[index["map"].isin(maps)]
|
| 333 |
+
vocab = KeyVocab(tuple(valid_keys) if valid_keys else CS2_KEYS, on_unknown="ignore")
|
| 334 |
+
ds = Cs2StreamingDataset(
|
| 335 |
+
index,
|
| 336 |
+
vocab=vocab,
|
| 337 |
+
clip_len=clip_len,
|
| 338 |
+
target_fps=target_fps,
|
| 339 |
+
n_players=n_players,
|
| 340 |
+
frame_size=frame_size,
|
| 341 |
+
shuffle=shuffle,
|
| 342 |
+
infinite=infinite,
|
| 343 |
+
seed=seed,
|
| 344 |
+
from_tar=(subset == "full"), # full split = ranged tar-member fetch; sample = individual files
|
| 345 |
+
)
|
| 346 |
+
return DataLoader(
|
| 347 |
+
ds,
|
| 348 |
+
batch_size=batch_size * n_players, # n_players contiguous rows per sample-block
|
| 349 |
+
num_workers=num_workers,
|
| 350 |
+
collate_fn=_collate,
|
| 351 |
+
drop_last=True,
|
| 352 |
+
prefetch_factor=prefetch_factor if num_workers else None,
|
| 353 |
+
)
|
world_model_030000.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:059bb8f777295197ad46f14ccf81b0d26bedcd76bddaa991dd7dd17beb1a0ee3
|
| 3 |
+
size 50158667
|