Quazim0t0 commited on
Commit
242cc21
·
verified ·
1 Parent(s): 6328ff4

CS2 MIRA-style world model: codec + WM weights, streaming code, model card

Browse files
.gitattributes CHANGED
@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ assets/codec_reconstruction.png filter=lfs diff=lfs merge=lfs -text
37
+ assets/real_vs_generated_final.png filter=lfs diff=lfs merge=lfs -text
38
+ assets/walkthrough.gif filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,131 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ tags:
4
+ - world-model
5
+ - video-generation
6
+ - diffusion
7
+ - flow-matching
8
+ - counter-strike
9
+ - reinforcement-learning-environment
10
+ library_name: pytorch
11
+ ---
12
+
13
+ # CS2 Latent World Model (MIRA-style)
14
+
15
+ An **action-conditioned world model for Counter-Strike 2** — feed it keyboard + mouse actions and it
16
+ imagines the next frames. Built by adapting [MIRA](https://github.com/mira-wm/mira) to stream
17
+ [RekaAI/CS2-10k](https://huggingface.co/datasets/RekaAI/CS2-10k) (63 TB) with **no local dataset copy**.
18
+
19
+ <p align="center">
20
+ <img src="assets/walkthrough.gif" alt="Walking around inside the world model" width="640"/>
21
+ </p>
22
+
23
+ *Driving the world model with "hold W + sweep the mouse" — every frame above is **generated**, not real gameplay.*
24
+
25
+ > **Status: research prototype.** It works end-to-end (coherent, action-responsive rollouts) but is
26
+ > small, blurry, and trained on very little data. Limitations are documented honestly below.
27
+
28
+ ## Real vs. generated
29
+
30
+ | Real gameplay (top) vs. world model (bottom) |
31
+ |---|
32
+ | ![real vs generated](assets/real_vs_generated_final.png) |
33
+
34
+ | Codec reconstruction — ground truth ‖ reconstruction |
35
+ |---|
36
+ | ![codec reconstruction](assets/codec_reconstruction.png) |
37
+
38
+ ## Architecture
39
+
40
+ **Stage 1 — Video codec (RAEv2)**
41
+ - Frozen **DINOv3-L/16** (`facebook/dinov3-vitl16-pretrain-lvd1689m`) backbone, layer aggregation
42
+ over blocks `[11,13,15,17,19,21,23]`.
43
+ - Strided-conv bottleneck → **32-channel latent at /32 spatial, /2 temporal**.
44
+ - ViT decoder (width 512, depth 6, patch 16). ~331 M params, mostly frozen.
45
+
46
+ **Stage 2 — Latent world model**
47
+ - Causal **flow-matching diffusion transformer** over codec latents — **12.5 M trainable** params
48
+ (hidden 384, 4 layers, 6 heads).
49
+ - AdaLN conditioning on **per-frame keyboard + mouse** actions; diffusion forcing; action dropout 0.1.
50
+ - Autoregressive rollout with a streaming KV-cache.
51
+
52
+ **IO:** 288×512 @ 24 fps, 16-frame clips (CS2 is 48 fps → integer /2 downsample).
53
+
54
+ ## Action space
55
+
56
+ 10 keys — `W A S D J C R V [ ]` (move, jump, crouch, walk, freefall, fire, scope) as a multi-hot
57
+ vector, **plus continuous mouse deltas**.
58
+
59
+ > Upstream MIRA conditions on *key presses only* (Rocket League is keyboard-only). CS2 is
60
+ > mouse-driven, so mouse deltas are populated here and consumed by MIRA's existing `ActionEncoder`
61
+ > (symlog-normalized + temporally pooled).
62
+
63
+ ## Files
64
+
65
+ | File | What |
66
+ |---|---|
67
+ | `codec_069000.pt` | Codec weights (decoder + bottleneck; frozen DINOv3 rebuilt from HF) |
68
+ | `world_model_030000.pt` | World model weights (12.5 M) |
69
+ | `scripts/`, `src/`, `configs/` | Training / eval / streaming code |
70
+ | `patches/dino_hf_backbone.patch` | Load gated DINOv3 via `transformers` instead of Meta's `.pth` |
71
+ | `modal_app.py` | Run the same training on a cloud GPU |
72
+
73
+ ## Training
74
+
75
+ | | Codec | World model |
76
+ |---|---|---|
77
+ | Steps | 69,000 | 30,000 |
78
+ | Hardware | RTX 5060 Ti (16 GB) + RTX PRO 6000 (Modal) | RTX 5060 Ti |
79
+ | Loss | L1 reconstruction | flow-matching diffusion |
80
+ | Final | ema 0.1392 | ema 2.41 |
81
+
82
+ Trained on the untarred `sample/` split: **3 matches / 374 clips / 44 (match, round) groups.**
83
+
84
+ ## Results
85
+
86
+ - **Codec reconstruction: 24.75 dB PSNR** (vs 19.7 dB with a random backbone). Recovers room
87
+ geometry and recognizable player figures; no fine texture.
88
+ - **Rollouts stay coherent** — players persist across the clip, room geometry is stable.
89
+ - **Controllability** — mean abs pixel difference of a generated rollout vs an idle-action rollout:
90
+ - `forward` vs `idle`: **0.025 – 0.037**
91
+ - `turn` vs `idle`: **0.040**
92
+ - Non-zero **and growing over the clip** → the model integrates actions over time rather than
93
+ ignoring them. This was the go/no-go test, and it passed.
94
+
95
+ ## Limitations
96
+
97
+ - **Blurry.** The /32, 32-channel latent + a depth-6 decoder caps detail (no faces, weapon detail,
98
+ floor texture). The paper uses a ViT-XL decoder.
99
+ - **Long-horizon drift.** Coherent for several frames past context, then degrades — dissolving to
100
+ generic wall texture or darkening. 16-frame clips vs the paper's 80.
101
+ - **Tiny dataset — 3 matches.** The dominant constraint; there's little to generalize from.
102
+ - **Tiny model** — 12.5 M vs the paper's 1 B. Long-horizon coherence is what capacity buys.
103
+ - **Single-perspective only.** MIRA's defining contribution is *multiplayer* (joint, time-aligned
104
+ multi-POV with per-player action attribution). Only the single-player stage is trained here. The
105
+ multi-POV data path exists (`cs2_stream` groups POVs by `(match_id, round_number)`, with ranged
106
+ tar-member fetch for the full split) but the joint stage was never trained.
107
+ - **Action semantics unverified.** We measure that actions *change* the output, not that "forward"
108
+ moves precisely forward — fidelity is too low to confirm direction visually.
109
+
110
+ ## Intended use / out of scope
111
+
112
+ Research on world models and action-conditioned video prediction. **Not** a game, an anti-cheat
113
+ signal, a player model, or anything deployable. Do not present generated frames as real gameplay.
114
+
115
+ ## Attribution & licensing
116
+
117
+ - **MIRA** — architecture/training code (upstream; this repo ships patches + new modules).
118
+ - **DINOv3** (Meta) — frozen backbone, **gated weights** under Meta's license (not redistributed here).
119
+ - **CS2-10k** (RekaAI) — **CC BY-NC 4.0**.
120
+ - Counter-Strike 2 imagery © Valve; generated frames derive from that footage.
121
+
122
+ **Non-commercial research use only.**
123
+
124
+ ## Usage
125
+
126
+ ```bash
127
+ export RS_DINO_HF=facebook/dinov3-vitl16-pretrain-lvd1689m # requires HF access to gated DINOv3
128
+ python scripts/consolidate_codec.py # pack codec_069000.pt for the world model
129
+ python scripts/train_cs2_wm.py --test # controllability check (generates rollouts)
130
+ python scripts/eval_cs2_codec.py # reconstruction PSNR + side-by-side frames
131
+ ```
assets/codec_reconstruction.png ADDED

Git LFS Details

  • SHA256: 35e55f93e6bd710c1f31f0f325d56c7b70aa28e433f465a35de7302668a5a99d
  • Pointer size: 131 Bytes
  • Size of remote file: 406 kB
assets/real_vs_generated_final.png ADDED

Git LFS Details

  • SHA256: c41a7cea9ea6d048b2443fb78e038f24ec102d2e8305dbb30b05c70c80d78c81
  • Pointer size: 132 Bytes
  • Size of remote file: 2.19 MB
assets/walkthrough.gif ADDED

Git LFS Details

  • SHA256: 34fd53b8389219d083bfd71d3533c83567b751702050943e7317a768af749297
  • Pointer size: 132 Bytes
  • Size of remote file: 3.2 MB
codec_069000.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:61001017c77e1921911f88696f9fd747e69a2005c4ca8c6d91afdc56d9b99842
3
+ size 111627229
configs/actions/cs2.yaml ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Counter-Strike 2 keybinds for CS2-10k (RekaAI). Per-frame `actions` string chars, stable order.
2
+ # `target_fps` must divide the 48fps source (compute_stride is integer): 24, 16, 12, 8, 6.
3
+ # Unlike keyboard-only Rocket League, CS2 also carries continuous mouse deltas
4
+ # (mouse_x_delta/mouse_y_delta), populated into ActionTensors.mouse_movements by cs2_stream.
5
+ target_fps: 24
6
+ valid_keys:
7
+ - W # forward
8
+ - A # left
9
+ - S # back
10
+ - D # right
11
+ - J # jump
12
+ - C # crouch
13
+ - R # walk (slow)
14
+ - V # freefall / release
15
+ - "[" # fire (attack)
16
+ - "]" # scope / secondary
modal_app.py ADDED
@@ -0,0 +1,103 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Run the CS2 MIRA training on Modal (e.g. an RTX PRO 6000 / H200, 90GB+ VRAM) to finish faster.
2
+
3
+ Same model config as local (nothing upgraded) — just a much faster GPU and no thermal pauses.
4
+ Checkpoints live in a Modal Volume so they persist and can be pulled back down.
5
+
6
+ SETUP (once, on your machine with the Modal CLI + `modal setup` done):
7
+ # 1. store your HF token (gated DINOv3 + CS2 dataset) as a Modal secret named "huggingface":
8
+ modal secret create huggingface HF_TOKEN=hf_xxxxxxxx
9
+ # 2. seed the volume with your latest local codec checkpoint so Modal RESUMES (not restarts):
10
+ modal volume create cs2-runs
11
+ modal volume put cs2-runs runs/cs2_codec_dino/codec_048000.pt cs2_codec_dino/codec_048000.pt
12
+
13
+ RUN (from the mira/ dir):
14
+ modal run modal_app.py # resume codec -> 120k on the cloud GPU
15
+ modal run modal_app.py --stage wm # (optional) train the small world model instead
16
+
17
+ PULL RESULTS BACK:
18
+ modal volume get cs2-runs cs2_codec_dino/codec_120000.pt ./runs/cs2_codec_dino/
19
+
20
+ Default GPU is the RTX PRO 6000 (96GB Blackwell). The config uses only ~6GB, so the VRAM is plenty;
21
+ raise CS2_BS to use more of it for speed. Override with MODAL_GPU=... if that string isn't accepted.
22
+ NOTE: RTX PRO 6000 is Blackwell (sm_120) — the image already pins torch cu132, which has sm_120
23
+ kernels (same reason we needed cu132 on your local 5060 Ti).
24
+ """
25
+ import os
26
+ import modal
27
+
28
+ GPU = os.environ.get("MODAL_GPU", "RTX-PRO-6000") # 96GB Blackwell; override via MODAL_GPU if needed
29
+
30
+ image = (
31
+ modal.Image.debian_slim(python_version="3.11")
32
+ .apt_install("git", "ffmpeg")
33
+ # torch cu132 = the build with Blackwell (sm_120) kernels, matching what we validated locally;
34
+ # it also runs fine on Hopper (H100/H200). Swap the index if your GPU needs a different CUDA.
35
+ .pip_install("torch==2.13.0", "torchvision==0.28.0",
36
+ index_url="https://download.pytorch.org/whl/cu132")
37
+ .pip_install(
38
+ "transformers<5", "huggingface_hub", "hf_transfer", "safetensors", "av", "pyarrow", "pandas",
39
+ "omegaconf", "pydantic", "einops", "torchmetrics", "termcolor", "fvcore", "iopath",
40
+ "pillow", "numpy", "tqdm", "rich",
41
+ )
42
+ # ship our code (cs2_stream, the HF DINOv3 adapter, the train scripts, the cs2 action config)
43
+ .add_local_dir("src", "/root/mira/src")
44
+ .add_local_dir("scripts", "/root/mira/scripts")
45
+ .add_local_dir("configs", "/root/mira/configs")
46
+ )
47
+
48
+ app = modal.App("cs2-mira", image=image)
49
+ runs = modal.Volume.from_name("cs2-runs", create_if_missing=True)
50
+ COMMON = dict(volumes={"/root/mira/runs": runs},
51
+ secrets=[modal.Secret.from_name("huggingface")],
52
+ timeout=24 * 3600)
53
+
54
+
55
+ def _run(script: str, extra_env: dict):
56
+ import subprocess
57
+ env = {**os.environ,
58
+ "RS_DINO_HF": "facebook/dinov3-vitl16-pretrain-lvd1689m", # real gated weights via HF
59
+ "HF_HUB_ENABLE_HF_TRANSFER": "1",
60
+ "PYTHONIOENCODING": "utf-8",
61
+ "PYTHONUNBUFFERED": "1",
62
+ **extra_env}
63
+ subprocess.run(["python", "-u", script], cwd="/root/mira", env=env, check=True)
64
+ runs.commit() # persist checkpoints written to the volume
65
+
66
+
67
+ @app.function(gpu=GPU, **COMMON)
68
+ def train_codec():
69
+ # identical config to local: T=16, 288x512@24fps, small decoder. Resumes from newest checkpoint
70
+ # already in the volume (upload your local one first). No thermal pauses on the cloud GPU.
71
+ _run("scripts/train_cs2_codec.py", {
72
+ "CS2_T": "16",
73
+ # NOTE: batch size does NOT reduce steps-to-finish — 150k steps @ bs64 is ~64x the compute of
74
+ # your 120k @ bs1 plan. To finish the SAME run faster, keep bs small and let the fast GPU do
75
+ # it. bs4 + workers keeps the decode pipeline fed without ballooning per-step cost.
76
+ "CS2_BS": os.environ.get("CS2_BS", "40"),
77
+ "CS2_WORKERS": os.environ.get("CS2_WORKERS", "8"), # Linux: parallel clip decode
78
+ "CS2_STEPS": os.environ.get("CS2_STEPS", "120000"),
79
+ "CS2_KEEP": "3",
80
+ "CS2_OUT": "runs/cs2_codec_dino",
81
+ "CS2_PAUSE_EVERY": "0",
82
+ })
83
+
84
+
85
+ @app.function(gpu=GPU, **COMMON)
86
+ def train_wm():
87
+ # the same tiny world model, on the consolidated codec in the volume.
88
+ _run("scripts/train_cs2_wm.py", {
89
+ "CS2_T": "16",
90
+ "CS2_WM_STEPS": os.environ.get("CS2_WM_STEPS", "20000"),
91
+ "CS2_WM_OUT": "runs/cs2_wm",
92
+ "CS2_CODEC": "runs/cs2_codec_consolidated/codec.pth",
93
+ })
94
+
95
+
96
+ @app.local_entrypoint()
97
+ def main(stage: str = "codec"):
98
+ if stage == "codec":
99
+ train_codec.remote()
100
+ elif stage == "wm":
101
+ train_wm.remote()
102
+ else:
103
+ raise SystemExit("stage must be 'codec' or 'wm'")
patches/dino_hf_backbone.patch ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ diff --git a/src/mira/codec/dino.py b/src/mira/codec/dino.py
2
+ index f2fd849..c0dbf25 100644
3
+ --- a/src/mira/codec/dino.py
4
+ +++ b/src/mira/codec/dino.py
5
+ @@ -62,6 +62,42 @@ DEFAULT_DINO_LAYERS = {
6
+ }
7
+
8
+
9
+ +class _HFDinoBackbone(nn.Module):
10
+ + """Adapter exposing the hub `get_intermediate_layers` API over a HuggingFace `transformers`
11
+ + DINOv3 model. Lets us use the real gated weights (e.g. facebook/dinov3-vitl16-pretrain-lvd1689m,
12
+ + which ship as HF safetensors) without the Meta `.pth` that torch.hub expects. Enabled by setting
13
+ + the env var RS_DINO_HF to the HF repo id."""
14
+ +
15
+ + def __init__(self, repo_id: str):
16
+ + super().__init__()
17
+ + from transformers import AutoModel
18
+ + self.model = AutoModel.from_pretrained(repo_id).eval()
19
+ + for p in self.model.parameters():
20
+ + p.requires_grad_(False)
21
+ + cfg = self.model.config
22
+ + self.n_prefix = 1 + int(getattr(cfg, "num_register_tokens", 0)) # cls + register tokens
23
+ + self.patch = int(getattr(cfg, "patch_size", 16))
24
+ + self.final_norm = getattr(self.model, "layernorm", None) or getattr(self.model, "norm", None)
25
+ +
26
+ + @torch.no_grad()
27
+ + def get_intermediate_layers(self, x, n, norm: bool = True, reshape: bool = True):
28
+ + out = self.model(pixel_values=x, output_hidden_states=True)
29
+ + hs = out.hidden_states # (num_blocks + 1) tensors, index 0 = embeddings
30
+ + n_blocks = len(hs) - 1
31
+ + idxs = tuple(n) if isinstance(n, (list, tuple)) else tuple(range(n_blocks - int(n), n_blocks))
32
+ + H, W = x.shape[-2] // self.patch, x.shape[-1] // self.patch
33
+ + feats = []
34
+ + for j in idxs:
35
+ + h = hs[j + 1] # output after block j
36
+ + if norm and self.final_norm is not None:
37
+ + h = self.final_norm(h)
38
+ + patch = h[:, self.n_prefix:, :] # drop cls + register tokens
39
+ + if reshape:
40
+ + patch = patch.reshape(patch.shape[0], H, W, patch.shape[-1]).permute(0, 3, 1, 2).contiguous()
41
+ + feats.append(patch)
42
+ + return feats
43
+ +
44
+ +
45
+ class DinoModel(nn.Module):
46
+ # The hub-loaded (or preloaded) backbone; typed loosely as its API is dynamic.
47
+ dino_model: Any
48
+ @@ -90,8 +126,12 @@ class DinoModel(nn.Module):
49
+ self.layers = DEFAULT_DINO_LAYERS[dino_model]
50
+
51
+ self.dino_dim = DINO_DIM[dino_model]
52
+ + _hf_repo = os.environ.get("RS_DINO_HF")
53
+ if preloaded_dino_module:
54
+ self.dino_model = preloaded_dino_module
55
+ + elif _hf_repo:
56
+ + logger.info(f"Loading DINOv3 via HuggingFace transformers: {_hf_repo}")
57
+ + self.dino_model = _HFDinoBackbone(_hf_repo)
58
+ else:
59
+ logging.getLogger("dinov3").setLevel(logging.WARNING) # suppress noisy dinov3 logging
60
+ logger.info(f"Loading DINOv3 model, variant {dino_model}, {compile=}")
scripts/consolidate_codec.py ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Consolidate our decoder-only CS2 codec checkpoint into the format MIRA's world model expects:
2
+ a full VideoCodec state_dict (real HF DINOv3 backbone + our trained decoder) + a codec_config.yaml
3
+ alongside, so LatentWorldModel can load it via VideoCodec.load_from_checkpoint.
4
+ """
5
+ import os, sys
6
+ os.environ.setdefault("RS_DINO_HF", "facebook/dinov3-vitl16-pretrain-lvd1689m")
7
+ sys.path.insert(0, "src")
8
+ import torch
9
+ from omegaconf import OmegaConf
10
+ from mira.codec import VideoCodec
11
+ from mira.codec.config import (
12
+ VideoCodecConfig, RAEEncoderConfig, ViTDecoderConfig, StridedConvBottleneckConfig, ImageConfig,
13
+ )
14
+
15
+ SRC = os.environ.get("CS2_CODEC_CKPT", "runs/cs2_codec_dino/codec_036000.pt")
16
+ OUT = os.environ.get("CS2_CODEC_OUT", "runs/cs2_codec_consolidated")
17
+ T, H, W = int(os.environ.get("CS2_T", "16")), 288, 512
18
+
19
+ img = ImageConfig(height=H, width=W, channels=3, timesteps=T, fps=24)
20
+ enc = RAEEncoderConfig(latent_dim=32, rae_model="dinov3_vitl16",
21
+ aggregation_layers=[11, 13, 15, 17, 19, 21, 23],
22
+ bottleneck=StridedConvBottleneckConfig(stride=2, temporal_stride=2, noise_tau=0.0),
23
+ compile_dino=False, video=img)
24
+ dec = ViTDecoderConfig(latent_dim=32, bottleneck=StridedConvBottleneckConfig(stride=2),
25
+ vit_width=512, vit_depth=6, vit_num_heads=8, mlp_dim_multiplier=4,
26
+ qk_norm="layernorm", patch_size=16, patch_size_t=2, video=img,
27
+ activation_checkpointing=True)
28
+ cfg = VideoCodecConfig(encoder=enc, decoder=dec)
29
+
30
+ print("[consolidate] building codec (real HF DINOv3 backbone) + loading trained decoder...", flush=True)
31
+ model = VideoCodec(cfg, require_dino_weights=False).eval()
32
+ sd = torch.load(SRC, map_location="cpu", weights_only=False)
33
+ missing, unexpected = model.load_state_dict(sd["model"], strict=False)
34
+ print(f"[consolidate] loaded decoder (missing={len(missing)} keys are the frozen backbone; "
35
+ f"unexpected={len(unexpected)})", flush=True)
36
+
37
+ os.makedirs(OUT, exist_ok=True)
38
+ OmegaConf.save(OmegaConf.create({"model": {"architecture": {"config": cfg.model_dump()}}}),
39
+ os.path.join(OUT, VideoCodec.CONFIG_FILENAME))
40
+ ckpt_path = os.path.join(OUT, "codec.pth")
41
+ torch.save({"state_dict": model.state_dict()}, ckpt_path)
42
+ print(f"[consolidate] wrote {ckpt_path} + {VideoCodec.CONFIG_FILENAME}", flush=True)
43
+
44
+ # verify round-trip through MIRA's loader
45
+ loaded = VideoCodec.load_from_checkpoint(ckpt_path, device="cpu")
46
+ print(f"[ok] round-trips via load_from_checkpoint: "
47
+ f"{sum(p.numel() for p in loaded.parameters())/1e6:.0f}M params, "
48
+ f"latent_dim={loaded.latent_dim}, td={loaded.temporal_downsampling}, "
49
+ f"sd={loaded.spatial_downsampling}", flush=True)
scripts/eval_cs2_codec.py ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Check codec rendering quality: load a checkpoint, reconstruct a streamed CS2 clip, and save
2
+ ground-truth | reconstruction frames side by side (+ PSNR). Point at a checkpoint via CS2_CKPT,
3
+ else the newest in runs/cs2_codec/.
4
+ """
5
+ import os, sys, glob, math
6
+ os.environ.setdefault("RS_DINO_HF", "facebook/dinov3-vitl16-pretrain-lvd1689m") # real backbone
7
+ sys.path.insert(0, "src")
8
+ import torch
9
+ from PIL import Image
10
+
11
+ from mira.codec import VideoCodec
12
+ from mira.codec.config import (
13
+ VideoCodecConfig, RAEEncoderConfig, ViTDecoderConfig, StridedConvBottleneckConfig, ImageConfig,
14
+ )
15
+ from mira.codec.viz import visualize_side_by_side
16
+ from mira.data.cs2_stream import create_cs2_loader
17
+
18
+ dev = "cuda" if torch.cuda.is_available() else "cpu"
19
+ T = int(os.environ.get("CS2_T", "24")); H, W, FPS = 288, 512, 24
20
+
21
+ # same architecture as training (weights, incl. frozen DINOv3, come from the checkpoint)
22
+ img = ImageConfig(height=H, width=W, channels=3, timesteps=T, fps=FPS)
23
+ enc = RAEEncoderConfig(latent_dim=32, rae_model="dinov3_vitl16",
24
+ aggregation_layers=[11, 13, 15, 17, 19, 21, 23],
25
+ bottleneck=StridedConvBottleneckConfig(stride=2, temporal_stride=2, noise_tau=0.0),
26
+ compile_dino=False, video=img)
27
+ dec = ViTDecoderConfig(latent_dim=32, bottleneck=StridedConvBottleneckConfig(stride=2),
28
+ vit_width=512, vit_depth=6, vit_num_heads=8, mlp_dim_multiplier=4,
29
+ qk_norm="layernorm", patch_size=16, patch_size_t=2, video=img,
30
+ activation_checkpointing=False)
31
+ model = VideoCodec(VideoCodecConfig(encoder=enc, decoder=dec), require_dino_weights=False).to(dev).eval()
32
+
33
+ ckpt = os.environ.get("CS2_CKPT") or max(glob.glob("runs/cs2_codec/*.pt"), key=os.path.getmtime)
34
+ sd = torch.load(ckpt, map_location=dev, weights_only=False)
35
+ model.load_state_dict(sd["model"], strict=False) # frozen backbone comes from HF, not the ckpt
36
+ print(f"[eval] loaded {ckpt} (step {sd.get('step')}, ema_loss {sd.get('ema_loss')})", flush=True)
37
+
38
+ loader = create_cs2_loader(subset=os.environ.get("CS2_SUBSET", "sample"), n_players=1, clip_len=T,
39
+ target_fps=FPS, frame_size=(H, W), batch_size=1, num_workers=0, infinite=True)
40
+ batch, meta = next(iter(loader))
41
+ with torch.no_grad():
42
+ out = model(batch.to(dev))
43
+
44
+ # PSNR over the whole clip (input/output are ~[-1,1] -> [0,1])
45
+ gt = (out.input_video * 0.5 + 0.5).clamp(0, 1)
46
+ pr = (out.output_video * 0.5 + 0.5).clamp(0, 1)
47
+ mse = torch.mean((gt - pr) ** 2).item()
48
+ psnr = 10 * math.log10(1.0 / mse) if mse > 0 else float("inf")
49
+ print(f"[eval] reconstruction PSNR = {psnr:.2f} dB (higher is better; ~20 rough, 28+ good)", flush=True)
50
+
51
+ viz = visualize_side_by_side(out)["viz_video"] # (B,T,C,H,2W) uint8, GT|recon
52
+ viz = viz[0] # (T,C,H,2W)
53
+ outdir = os.path.join(os.path.dirname(ckpt), "recon"); os.makedirs(outdir, exist_ok=True)
54
+ saved = []
55
+ for i in {0, T // 2, T - 1}:
56
+ im = viz[i].permute(1, 2, 0).cpu().numpy() # (H, 2W, C)
57
+ p = os.path.join(outdir, f"step{sd.get('step')}_frame{i:02d}_gt-left_recon-right.png")
58
+ Image.fromarray(im).save(p); saved.append(p)
59
+ print("[eval] saved (left = ground truth, right = reconstruction):", flush=True)
60
+ for p in saved:
61
+ print(" ", p, flush=True)
scripts/train_cs2_codec.py ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Minimal codec training on STREAMED CS2-10k — proves the streamed data -> MIRA model -> loss
2
+ pipeline end to end, in the existing venv (no pixi, no hydra). Pure L1 reconstruction so it needs
3
+ NO external weights; `require_dino_weights=False` builds the DINOv3 backbone with random weights
4
+ (swap in real gated DINOv3 later for a real RAE codec). Small config to fit 16GB and run fast.
5
+ """
6
+ import os, sys, time, traceback
7
+ sys.path.insert(0, "src")
8
+ import torch
9
+
10
+ from mira.codec import VideoCodec, CodecLoss
11
+ from mira.codec.config import (
12
+ VideoCodecConfig, RAEEncoderConfig, ViTDecoderConfig,
13
+ StridedConvBottleneckConfig, ImageConfig,
14
+ )
15
+ from mira.codec.loss import CodecLossWeights
16
+ from mira.data.cs2_stream import create_cs2_loader
17
+
18
+ dev = "cuda" if torch.cuda.is_available() else "cpu"
19
+ # MIRA codec specs (paper Sec 6.3 / Table 9): per-view 288x512, 40-frame clips, DINOv3-L backbone,
20
+ # 1 latent per 32x32 px at half the input rate. CS2 is 48fps -> 24fps (48 isn't divisible by 20).
21
+ # Env knobs for 16GB tuning: CS2_T (frames, default 40), CS2_BS (batch, default 1), CS2_SUBSET.
22
+ T = int(os.environ.get("CS2_T", "40"))
23
+ FPS = 24
24
+ H, W = 288, 512
25
+ img = ImageConfig(height=H, width=W, channels=3, timesteps=T, fps=FPS)
26
+ enc = RAEEncoderConfig(
27
+ latent_dim=32, rae_model="dinov3_vitl16",
28
+ aggregation_layers=[11, 13, 15, 17, 19, 21, 23],
29
+ bottleneck=StridedConvBottleneckConfig(stride=2, temporal_stride=2, noise_tau=0.0),
30
+ compile_dino=False, video=img,
31
+ )
32
+ dec = ViTDecoderConfig(
33
+ latent_dim=32, bottleneck=StridedConvBottleneckConfig(stride=2),
34
+ # decoder size from env (defaults small/16GB; paper ViT-XL ~ 1152/28/16 on big GPUs)
35
+ vit_width=int(os.environ.get("CODEC_VIT_WIDTH", "512")),
36
+ vit_depth=int(os.environ.get("CODEC_VIT_DEPTH", "6")),
37
+ vit_num_heads=int(os.environ.get("CODEC_VIT_HEADS", "8")),
38
+ mlp_dim_multiplier=4,
39
+ qk_norm="layernorm", patch_size=16, patch_size_t=2, video=img, # 16*bottleneck_stride(2)=32 == encoder /32
40
+ activation_checkpointing=True,
41
+ )
42
+ cfg = VideoCodecConfig(encoder=enc, decoder=dec)
43
+
44
+ # Real gated DINOv3-L weights are used iff RS_DINO_WEIGHTS_DIR points at them (download from Meta:
45
+ # dinov3_vitl16_pretrain_lvd1689m-*.pth). Otherwise fall back to a random backbone so it still runs.
46
+ want_dino = bool(os.environ.get("RS_DINO_WEIGHTS_DIR"))
47
+ print(f"[cs2-codec] device={dev} T={T} {H}x{W}@{FPS}fps real_dino_weights={want_dino}", flush=True)
48
+ try:
49
+ model = VideoCodec(cfg, require_dino_weights=want_dino).to(dev).train()
50
+ except FileNotFoundError:
51
+ print("[cs2-codec] RS_DINO_WEIGHTS_DIR set but weights not found; using RANDOM backbone. "
52
+ "For a real codec, put dinov3_vitl16_pretrain_lvd1689m-*.pth in that dir.", flush=True)
53
+ model = VideoCodec(cfg, require_dino_weights=False).to(dev).train()
54
+ except Exception:
55
+ traceback.print_exc(); sys.exit(3)
56
+
57
+ print(f"[cs2-codec] params: {sum(p.numel() for p in model.parameters())/1e6:.1f}M", flush=True)
58
+ loss_fn = CodecLoss(CodecLossWeights(
59
+ loss_mae=1.0, loss_lpips_perceptual=0.0, loss_dino_latent_consistency=0.0, auto_weight=False,
60
+ )).to(dev)
61
+ opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
62
+
63
+ loader = create_cs2_loader(
64
+ subset=os.environ.get("CS2_SUBSET", "sample"), n_players=1, clip_len=T, target_fps=FPS,
65
+ frame_size=(H, W), batch_size=int(os.environ.get("CS2_BS", "1")),
66
+ num_workers=int(os.environ.get("CS2_WORKERS", "0")), # >0 on Linux (Modal) to parallelize decode
67
+ infinite=True,
68
+ )
69
+ print("[cs2-codec] streaming CS2 sample split; first batch downloads a few clips...", flush=True)
70
+
71
+ from pathlib import Path
72
+ STEPS = int(os.environ.get("CS2_STEPS", "20000"))
73
+ CKPT_EVERY = int(os.environ.get("CS2_CKPT_EVERY", "500"))
74
+ ckpt_dir = Path(os.environ.get("CS2_OUT", "runs/cs2_codec"))
75
+ ckpt_dir.mkdir(parents=True, exist_ok=True)
76
+ print(f"[cs2-codec] REAL RUN: {STEPS} steps, checkpoint every {CKPT_EVERY} -> {ckpt_dir}", flush=True)
77
+
78
+ # Resume from the newest checkpoint (or CS2_RESUME) so restarts don't lose progress.
79
+ import glob
80
+ _start_step, _ema0 = 0, None
81
+ _resume = os.environ.get("CS2_RESUME") or (sorted(glob.glob(str(ckpt_dir / "codec_0*.pt"))) or [None])[-1]
82
+ def _trainable_sd():
83
+ # exclude the frozen DINOv3 backbone (reloaded from HF/hub) -> small, portable checkpoints
84
+ return {k: v for k, v in model.state_dict().items() if "rae_dino.dino_model" not in k}
85
+
86
+ if _resume and os.path.exists(_resume):
87
+ _sd = torch.load(_resume, map_location=dev, weights_only=False)
88
+ model.load_state_dict(_sd["model"], strict=False) # backbone already loaded (HF/hub)
89
+ _start_step = int(_sd.get("step", 0)); _ema0 = _sd.get("ema_loss")
90
+ print(f"[cs2-codec] resumed from {_resume} at step {_start_step} (ema_loss={_ema0})", flush=True)
91
+
92
+ # M.2 thermal duty-cycle: every CS2_PAUSE_EVERY steps, sleep CS2_PAUSE_SECS to let the SSD cool.
93
+ PAUSE_EVERY = int(os.environ.get("CS2_PAUSE_EVERY", "1000"))
94
+ PAUSE_SECS = int(os.environ.get("CS2_PAUSE_SECS", "900"))
95
+
96
+ step, ema = _start_step, _ema0
97
+ _pace_t0, _pace_s0 = time.time(), _start_step
98
+ for batch, meta in loader:
99
+ batch = batch.to(dev)
100
+ out = model(batch)
101
+ losses = loss_fn(out, global_step=step)
102
+ total = losses["loss"] if "loss" in losses else sum(v for v in losses.values() if v.ndim == 0)
103
+ opt.zero_grad(); total.backward(); opt.step()
104
+ ema = float(total) if ema is None else 0.98 * ema + 0.02 * float(total)
105
+ if step % 20 == 0:
106
+ print(f"[cs2-codec] step {step:6d} loss={float(total):.4f} ema={ema:.4f} "
107
+ f"({(time.time()-_pace_t0)/(step-_pace_s0+1):.2f}s/step)", flush=True)
108
+ step += 1
109
+ if step % CKPT_EVERY == 0:
110
+ torch.save({"step": step, "model": _trainable_sd(), "config": cfg.model_dump(),
111
+ "ema_loss": ema}, ckpt_dir / f"codec_{step:06d}.pt")
112
+ print(f"[cs2-codec] saved {ckpt_dir / f'codec_{step:06d}.pt'} (ema_loss={ema:.4f})", flush=True)
113
+ keep = int(os.environ.get("CS2_KEEP", "2")) # cap disk: keep only the newest N
114
+ for old in sorted(ckpt_dir.glob("codec_0*.pt"))[:-keep]:
115
+ old.unlink(missing_ok=True)
116
+ if step >= STEPS:
117
+ break
118
+ if PAUSE_EVERY and step % PAUSE_EVERY == 0: # checkpoint just saved (1000%500==0)
119
+ import datetime as _dt
120
+ print(f"[cs2-codec] PAUSE {PAUSE_SECS}s at step {step} for M.2 cooldown "
121
+ f"(resume ~{(_dt.datetime.now()+_dt.timedelta(seconds=PAUSE_SECS)).strftime('%H:%M:%S')})",
122
+ flush=True)
123
+ time.sleep(PAUSE_SECS)
124
+ _pace_t0, _pace_s0 = time.time(), step # exclude the sleep from the pace estimate
125
+ print(f"[cs2-codec] RESUME at step {step}", flush=True)
126
+ torch.save({"step": step, "model": _trainable_sd(), "config": cfg.model_dump(), "ema_loss": ema},
127
+ ckpt_dir / "codec_final.pt")
128
+ print(f"[cs2-codec] DONE {step} steps, final ema_loss={ema:.4f} -> {ckpt_dir/'codec_final.pt'}", flush=True)
scripts/train_cs2_wm.py ADDED
@@ -0,0 +1,121 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Train a small MIRA latent world model on the CS2 codec, and test action-controllability.
2
+
3
+ train: python scripts/train_cs2_wm.py
4
+ test : python scripts/train_cs2_wm.py --test (seed real context, feed fixed actions, generate)
5
+
6
+ Frozen codec is loaded from the consolidated checkpoint; a tiny flow-matching diffusion transformer
7
+ learns to predict the next codec latent conditioned on CS2 keyboard+mouse actions.
8
+ """
9
+ import os, sys, time, argparse
10
+ os.environ.setdefault("RS_DINO_HF", "facebook/dinov3-vitl16-pretrain-lvd1689m")
11
+ sys.path.insert(0, "src")
12
+ import torch
13
+
14
+ from mira.world_model.config import LatentWorldModelConfig
15
+ from mira.world_model.latent_world_model import LatentWorldModel
16
+ from mira.world_model.actions_config import ActionConfig
17
+ from mira.ml.image_config import ImageConfig
18
+ from mira.data.cs2_stream import create_cs2_loader, CS2_KEYS
19
+
20
+ dev = "cuda" if torch.cuda.is_available() else "cpu"
21
+ T = int(os.environ.get("CS2_T", "16"))
22
+ CODEC = os.environ.get("CS2_CODEC", "runs/cs2_codec_consolidated/codec.pth")
23
+ OUT = os.environ.get("CS2_WM_OUT", "runs/cs2_wm")
24
+
25
+
26
+ def build_model():
27
+ actions = ActionConfig(valid_keys=list(CS2_KEYS), source_fps=24, target_fps=24)
28
+ video = ImageConfig(height=288, width=512, channels=3, timesteps=T, fps=24)
29
+ cfg = LatentWorldModelConfig(
30
+ actions=actions, video=video, codec_checkpoint=CODEC,
31
+ latent_mean_std=[0.0, 1.0], # identity norm (we didn't estimate latent stats)
32
+ causal=True, use_clean_past=True, learned_temporal_pool=True,
33
+ use_codec_posterior_mean=True, attention_gating=True, ada_attn_ln=True,
34
+ patch_size=1, n_register_tokens=0, n_context_frames=max(2, T // 2 - 2),
35
+ dropout_action_prob=0.1,
36
+ # size from env (defaults = tiny/16GB-safe; scale up on big GPUs, e.g. 1B: 2048/16/4/16)
37
+ hidden_dim=int(os.environ.get("WM_HIDDEN", "384")),
38
+ n_head=int(os.environ.get("WM_HEADS", "6")),
39
+ n_kv_head=int(os.environ.get("WM_KV", "2")),
40
+ n_layers=int(os.environ.get("WM_LAYERS", "4")),
41
+ time_attention_every=int(os.environ.get("WM_TATT", "2")),
42
+ activation_checkpointing=os.environ.get("WM_ACT_CKPT", "0") == "1",
43
+ )
44
+ return LatentWorldModel(cfg).to(dev)
45
+
46
+
47
+ def train():
48
+ from pathlib import Path
49
+ model = build_model().train()
50
+ n_tr = sum(p.numel() for p in model.parameters() if p.requires_grad) / 1e6
51
+ print(f"[wm] trainable params: {n_tr:.1f}M (+ frozen codec)", flush=True)
52
+ opt = torch.optim.AdamW([p for p in model.parameters() if p.requires_grad], lr=1e-4)
53
+ loader = create_cs2_loader(subset=os.environ.get("CS2_SUBSET", "sample"), n_players=1, clip_len=T,
54
+ target_fps=24, frame_size=(288, 512),
55
+ batch_size=int(os.environ.get("WM_BATCH", "1")),
56
+ num_workers=int(os.environ.get("CS2_WORKERS", "0")), infinite=True)
57
+ ckpt_dir = Path(OUT); ckpt_dir.mkdir(parents=True, exist_ok=True)
58
+ STEPS = int(os.environ.get("CS2_WM_STEPS", "5000"))
59
+ step, ema, t0 = 0, None, time.time()
60
+ for batch, meta in loader:
61
+ losses = model(batch.to(dev))
62
+ total = losses["loss_total"]
63
+ opt.zero_grad(); total.backward(); opt.step()
64
+ ema = float(total) if ema is None else 0.98 * ema + 0.02 * float(total)
65
+ if step % 20 == 0:
66
+ print(f"[wm] step {step:5d} loss={float(total):.4f} ema={ema:.4f} "
67
+ f"({(time.time()-t0)/(step+1):.2f}s/step)", flush=True)
68
+ step += 1
69
+ if step % 500 == 0:
70
+ # exclude the frozen codec (rebuilt from CS2_CODEC) -> ~50MB instead of ~1.3GB
71
+ wm_only = {k: v for k, v in model.state_dict().items() if not k.startswith("codec.")}
72
+ torch.save({"step": step, "model": wm_only, "ema": ema}, ckpt_dir / f"wm_{step:06d}.pt")
73
+ for old in sorted(ckpt_dir.glob("wm_0*.pt"))[:-2]:
74
+ old.unlink(missing_ok=True)
75
+ print(f"[wm] saved wm_{step:06d}.pt (ema={ema:.4f})", flush=True)
76
+ if step >= STEPS:
77
+ break
78
+ print("[wm] done", flush=True)
79
+
80
+
81
+ @torch.no_grad()
82
+ def test():
83
+ """Seed a real context clip, then generate under three fixed action regimes and save frames."""
84
+ import glob, numpy as np
85
+ from PIL import Image
86
+ from mira.world_model.latent_world_model import WorldModelInferenceConfig
87
+ model = build_model().eval()
88
+ ck = os.environ.get("CS2_WM_CKPT") or (sorted(glob.glob(f"{OUT}/wm_0*.pt")) or [None])[-1]
89
+ if ck:
90
+ model.load_state_dict(torch.load(ck, map_location=dev, weights_only=False)["model"], strict=False)
91
+ print(f"[wm-test] loaded {ck}", flush=True)
92
+ loader = create_cs2_loader(subset="sample", n_players=1, clip_len=T, target_fps=24,
93
+ frame_size=(288, 512), batch_size=1, num_workers=0, infinite=True)
94
+ batch, _ = next(iter(loader))
95
+
96
+ # key indices in the CS2 vocab
97
+ kW, kMOUSE = CS2_KEYS.index("W"), None
98
+ regimes = {
99
+ "idle": lambda a: (a.key_presses.zero_(), a.mouse_movements.zero_()),
100
+ "forward": lambda a: (a.key_presses.zero_().__setitem__((slice(None), slice(None), kW), 1),
101
+ a.mouse_movements.zero_()),
102
+ "turn_right": lambda a: (a.key_presses.zero_(),
103
+ a.mouse_movements.zero_().__setitem__((slice(None), slice(None), 0), 25.0)),
104
+ }
105
+ icfg = WorldModelInferenceConfig() if False else None
106
+ outdir = os.path.join(OUT, "control"); os.makedirs(outdir, exist_ok=True)
107
+ for name, override in regimes.items():
108
+ b = batch.clone()
109
+ override(b.actions)
110
+ out = model.inference(b.to(dev), progress_bar=False)
111
+ vid = out.output_video[0] # (T,C,H,W) in [-1,1]
112
+ img = ((vid[-1] * 0.5 + 0.5).clamp(0, 1) * 255).byte().permute(1, 2, 0).cpu().numpy()
113
+ p = os.path.join(outdir, f"gen_{name}_lastframe.png")
114
+ Image.fromarray(img).save(p)
115
+ print(f"[wm-test] {name}: saved {p}", flush=True)
116
+ print("[wm-test] compare the three frames: forward/turn should differ from idle if controllable.", flush=True)
117
+
118
+
119
+ if __name__ == "__main__":
120
+ ap = argparse.ArgumentParser(); ap.add_argument("--test", action="store_true"); a = ap.parse_args()
121
+ (test if a.test else train)()
src/mira/data/cs2_stream.py ADDED
@@ -0,0 +1,353 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Streaming CS2-10k dataset for MIRA (no local dataset build).
2
+
3
+ RekaAI/CS2-10k is a WebDataset of first-person CS2 rounds. `index.parquet` lists every clip with:
4
+ match_id, video_path, parquet_path, map, player_index, round_number, team, fps(48), total_time,
5
+ width, height, match_num_clips, match_num_rounds, match_num_players, shard
6
+ Grouping key for MIRA's multi-perspective samples is (match_id, round_number): fixing both and
7
+ varying player_index gives up to `match_num_players` (=10) synchronized first-person views of the
8
+ same round — the CS2 analogue of MIRA's 4 Rocket League views.
9
+
10
+ This module streams those groups straight from the Hub and yields MIRA's own
11
+ `(VideoActionBatch, list[ClipMeta])` batches, so it drops into MIRA's trainer in place of
12
+ `create_loader` with zero on-disk dataset. It reuses MIRA's `decode_frames`, `KeyVocab`,
13
+ `ActionTensors`, `ClipMeta`, `VideoActionBatch`, and `_collate`.
14
+
15
+ Two important, honest caveats:
16
+ * ALIGNMENT: CS2 gives no shared per-frame tick. Each round's per-player clips are 48fps and
17
+ (empirically) start at round start, so we align at frame 0 and truncate a group to the shortest
18
+ clip. Refine later with position/rotation cross-checks if needed.
19
+ * SCALE / STREAMING UNIT: the 63TB main split is tar-sharded and a round's POVs are scattered
20
+ across many shards, so complete groups are NOT co-located in one tar. This loader therefore
21
+ fetches clips as INDIVIDUAL files by path (works today for the untarred `sample/` split — 3 full
22
+ matches — and for any individual-file mirror). Training on the full tarred split at scale needs
23
+ either re-sharding by (match,round) or a per-group multi-shard fetch; that's a separate step.
24
+ """
25
+
26
+ from __future__ import annotations
27
+
28
+ import io
29
+ import random
30
+ from collections import defaultdict
31
+ from itertools import count
32
+ from typing import Any, Iterator
33
+
34
+ import numpy as np
35
+ import torch
36
+ from torch.utils.data import DataLoader, IterableDataset, get_worker_info
37
+
38
+ from huggingface_hub import hf_hub_download
39
+ import pandas as pd
40
+
41
+ import av # ffmpeg-backed decode (works in this venv; avoids the torchcodec dependency)
42
+
43
+ from .actions import KeyVocab
44
+ from .batch import VideoActionBatch
45
+ from .clips import compute_stride
46
+ from .training_loader import ClipMeta, _collate # reuse MIRA's collate + meta type
47
+
48
+
49
+ def _decode_av(mp4_bytes: bytes, frame_indices: list[int], frame_size: tuple[int, int] | None):
50
+ """Decode the given source-frame indices from mp4 bytes to (T, C, H, W) uint8, via PyAV."""
51
+ want = set(frame_indices)
52
+ hi = max(frame_indices)
53
+ grabbed: dict[int, np.ndarray] = {}
54
+ with av.open(io.BytesIO(mp4_bytes)) as container:
55
+ for i, frame in enumerate(container.decode(container.streams.video[0])):
56
+ if i in want:
57
+ grabbed[i] = frame.to_ndarray(format="rgb24") # (H, W, 3) uint8
58
+ if i >= hi:
59
+ break
60
+ arr = np.stack([grabbed[i] for i in frame_indices]) # (T, H, W, 3)
61
+ t = torch.from_numpy(arr).permute(0, 3, 1, 2).contiguous() # (T, C, H, W) uint8
62
+ if frame_size is not None:
63
+ t = torch.nn.functional.interpolate(
64
+ t.float(), size=frame_size, mode="bilinear", align_corners=False
65
+ ).clamp(0, 255).to(torch.uint8)
66
+ return t
67
+
68
+ # CS2 held-key characters (per-frame `actions` string), stable multi-hot order. '-' == no input.
69
+ CS2_KEYS: tuple[str, ...] = ("W", "A", "S", "D", "J", "C", "R", "V", "[", "]")
70
+
71
+ REPO_ID = "RekaAI/CS2-10k"
72
+
73
+
74
+ def _rank_world() -> tuple[int, int]:
75
+ if torch.distributed.is_available() and torch.distributed.is_initialized():
76
+ return torch.distributed.get_rank(), torch.distributed.get_world_size()
77
+ return 0, 1
78
+
79
+
80
+ def _fetch_bytes(path: str) -> bytes:
81
+ """Fetch an INDIVIDUAL file by repo path (works for the untarred `sample/` split)."""
82
+ local = hf_hub_download(REPO_ID, path, repo_type="dataset")
83
+ with open(local, "rb") as f:
84
+ return f.read()
85
+
86
+
87
+ # --- ranged tar-member fetch for the FULL (tarred) split ------------------------------------------
88
+ # A round's POVs are scattered across many ~2GB tars. We open each tar over HfFileSystem (a seekable,
89
+ # HTTP-range-backed file) with tarfile mode="r", which walks the member headers via small ranged
90
+ # reads ONCE per shard, then extracts a single member with one ranged read of just its bytes — so we
91
+ # never download a whole 2GB tar. Open TarFiles are cached per shard.
92
+ import tarfile
93
+
94
+ _HF_FS = None
95
+ _TAR_CACHE: dict[str, tuple] = {}
96
+
97
+
98
+ def _hf_fs():
99
+ global _HF_FS
100
+ if _HF_FS is None:
101
+ from huggingface_hub import HfFileSystem
102
+ _HF_FS = HfFileSystem()
103
+ return _HF_FS
104
+
105
+
106
+ def _open_shard(shard: str):
107
+ if shard not in _TAR_CACHE:
108
+ f = _hf_fs().open(f"datasets/{REPO_ID}/{shard}", "rb") # seekable, range-backed
109
+ tf = tarfile.open(fileobj=f, mode="r") # header walk (ranged) once
110
+ members = {m.name.rsplit("/", 1)[-1]: m for m in tf.getmembers()}
111
+ if len(_TAR_CACHE) > 24: # bound open handles
112
+ old = next(iter(_TAR_CACHE))
113
+ try:
114
+ _TAR_CACHE.pop(old)[0].close()
115
+ except Exception:
116
+ pass
117
+ _TAR_CACHE[shard] = (tf, members)
118
+ return _TAR_CACHE[shard]
119
+
120
+
121
+ def _fetch_from_tar(shard: str, repo_path: str) -> bytes:
122
+ tf, members = _open_shard(shard)
123
+ m = members[repo_path.rsplit("/", 1)[-1]]
124
+ return tf.extractfile(m).read()
125
+
126
+
127
+ def _build_action_arrays(
128
+ frame_data: np.ndarray, vocab: KeyVocab, stride: int
129
+ ) -> tuple[torch.Tensor, torch.Tensor]:
130
+ """From a clip's 48fps per-frame records build downsampled (n_steps, n_keys) int32 multi-hot key
131
+ presses (OR-ed over each stride window) and (n_steps, 2) float32 mean mouse deltas."""
132
+ n_keys = len(vocab)
133
+ n_steps = len(frame_data) // stride
134
+ keys = torch.zeros((n_steps, n_keys), dtype=torch.int32)
135
+ mouse = torch.zeros((n_steps, 2), dtype=torch.float32)
136
+ for s in range(n_steps):
137
+ mx = my = 0.0
138
+ for j in range(s * stride, (s + 1) * stride):
139
+ rec = frame_data[j]
140
+ a = rec["actions"]
141
+ if a and a != "-":
142
+ for ch in a:
143
+ idx = vocab._index.get(ch)
144
+ if idx is not None:
145
+ keys[s, idx] = 1
146
+ mx += float(rec["mouse_x_delta"])
147
+ my += float(rec["mouse_y_delta"])
148
+ mouse[s, 0] = mx / stride
149
+ mouse[s, 1] = my / stride
150
+ return keys, mouse
151
+
152
+
153
+ class Cs2StreamingDataset(IterableDataset):
154
+ """Streams (match_id, round_number) POV groups from CS2-10k as MIRA per-perspective samples.
155
+
156
+ Yields dicts shaped exactly like MIRA's training_loader `_decode_sample` output
157
+ ({"video","actions","metadata"}), so `_collate` turns a batch into `(VideoActionBatch, [ClipMeta])`.
158
+ """
159
+
160
+ def __init__(
161
+ self,
162
+ index: pd.DataFrame,
163
+ *,
164
+ vocab: KeyVocab,
165
+ clip_len: int = 16,
166
+ target_fps: int = 16,
167
+ source_fps: int = 48,
168
+ n_players: int = 4,
169
+ frame_size: tuple[int, int] | None = None,
170
+ mouse_sensitivity: float = 1.0,
171
+ shuffle: bool = True,
172
+ infinite: bool = True,
173
+ seed: int = 2025,
174
+ from_tar: bool = False,
175
+ ) -> None:
176
+ super().__init__()
177
+ self.index = index
178
+ self.from_tar = from_tar
179
+ self.vocab = vocab
180
+ self.clip_len = clip_len
181
+ self.target_fps = target_fps
182
+ self.source_fps = source_fps
183
+ self.n_players = n_players
184
+ self.frame_size = frame_size
185
+ self.mouse_sensitivity = mouse_sensitivity
186
+ self.shuffle = shuffle
187
+ self.infinite = infinite
188
+ self.seed = seed
189
+ self.stride = compute_stride(source_fps, target_fps)
190
+ # group rows into (match_id, round_number) -> [row, ...] sorted by player_index
191
+ groups: dict[tuple[str, int], list[dict]] = defaultdict(list)
192
+ for r in index.to_dict("records"):
193
+ groups[(r["match_id"], int(r["round_number"]))].append(r)
194
+ # keep only groups with enough perspectives to form an n_players block
195
+ self.groups: list[list[dict]] = [
196
+ sorted(v, key=lambda r: r["player_index"])
197
+ for v in groups.values()
198
+ if len(v) >= n_players
199
+ ]
200
+
201
+ def _my_groups(self) -> list[list[dict]]:
202
+ rank, world = _rank_world()
203
+ info = get_worker_info()
204
+ wid, nw = (info.id, info.num_workers) if info else (0, 1)
205
+ return self.groups[rank::world][wid::nw]
206
+
207
+ def _load_group(self, rows: list[dict]) -> tuple[list[bytes], list[np.ndarray], list[dict]]:
208
+ import io as _io
209
+ vids, fdata, meta = [], [], []
210
+ for r in rows[: self.n_players]:
211
+ if self.from_tar: # full split: ranged tar members
212
+ vids.append(_fetch_from_tar(r["shard"], r["video_path"]))
213
+ pq = pd.read_parquet(_io.BytesIO(_fetch_from_tar(r["shard"], r["parquet_path"])))
214
+ else: # sample split: individual files
215
+ vids.append(_fetch_bytes(r["video_path"]))
216
+ pq = pd.read_parquet(hf_hub_download(REPO_ID, r["parquet_path"], repo_type="dataset"))
217
+ fdata.append(pq.iloc[0]["frame_data"])
218
+ meta.append(r)
219
+ return vids, fdata, meta
220
+
221
+ def _samples_from_group(self, rows: list[dict]) -> Iterator[dict[str, Any]]:
222
+ vids, fdata, meta = self._load_group(rows)
223
+ n_src = min(len(fd) for fd in fdata) # align at frame 0, truncate to shortest
224
+ n_steps = n_src // self.stride
225
+ if n_steps < self.clip_len:
226
+ return
227
+ # per-perspective full-round action arrays, then slice per clip window
228
+ key_arrs, mouse_arrs = [], []
229
+ for fd in fdata:
230
+ k, m = _build_action_arrays(fd[: n_steps * self.stride], self.vocab, self.stride)
231
+ key_arrs.append(k)
232
+ mouse_arrs.append(m)
233
+ # Decode every target-fps frame of the clip ONCE per perspective, then slice windows from it.
234
+ # (Decoding per window re-decoded from the start of the mp4 each time -> O(n^2) and starved
235
+ # the GPU; this is a single linear pass.)
236
+ all_idx = [s * self.stride for s in range(n_steps)]
237
+ full = [_decode_av(vids[p], all_idx, self.frame_size) for p in range(self.n_players)]
238
+ clip_id = 0
239
+ for start in range(0, n_steps - self.clip_len + 1, self.clip_len):
240
+ frame_indices = all_idx[start : start + self.clip_len]
241
+ for p in range(self.n_players):
242
+ video = full[p][start : start + self.clip_len] # (T,C,H,W) uint8, already decoded
243
+ act = _make_action_tensors(
244
+ key_arrs[p][start : start + self.clip_len],
245
+ mouse_arrs[p][start : start + self.clip_len],
246
+ self.vocab,
247
+ self.mouse_sensitivity,
248
+ )
249
+ yield {
250
+ "video": video,
251
+ "actions": act,
252
+ "metadata": ClipMeta(
253
+ match_id=meta[p]["match_id"],
254
+ perspective=p,
255
+ player_id=int(meta[p]["player_index"]),
256
+ clip_id=clip_id,
257
+ chunk_idx=int(meta[p]["round_number"]),
258
+ frame_indices=list(frame_indices),
259
+ ),
260
+ }
261
+ clip_id += 1
262
+
263
+ def __iter__(self) -> Iterator[dict[str, Any]]:
264
+ rank, _ = _rank_world()
265
+ info = get_worker_info()
266
+ rng = random.Random(self.seed + rank * 1024 + (info.id if info else 0))
267
+ for _epoch in count() if self.infinite else range(1):
268
+ order = self._my_groups()
269
+ if self.shuffle:
270
+ rng.shuffle(order)
271
+ for rows in order:
272
+ try:
273
+ yield from self._samples_from_group(rows)
274
+ except Exception as e: # a bad clip shouldn't kill the epoch
275
+ print(f"[cs2_stream] skipping group {rows[0]['match_id']}: {e}")
276
+
277
+
278
+ def _make_action_tensors(keys: torch.Tensor, mouse: torch.Tensor, vocab: KeyVocab, sens: float):
279
+ """Build a MIRA ActionTensors (batch=1) from a clip window's key/mouse tensors."""
280
+ from mira.world_model.actions_config import ActionConfig, ActionTensors
281
+
282
+ cfg = ActionConfig(valid_keys=list(vocab.keys))
283
+ at = ActionTensors(config=cfg, batch_size=1)
284
+ at.key_presses = keys.unsqueeze(0).to(torch.int32) # (1, T, n_keys)
285
+ at.mouse_movements = mouse.unsqueeze(0).to(torch.float32) # (1, T, 2)
286
+ at.game_mouse_sensitivity = torch.full((1,), float(sens), dtype=torch.float32)
287
+ return at
288
+
289
+
290
+ def load_cs2_index(subset: str = "sample") -> pd.DataFrame:
291
+ """Return the CS2 index as a DataFrame. `subset='sample'` uses the untarred 3-match `sample/`
292
+ split whose clips are individually fetchable (recommended for streaming today); `subset='full'`
293
+ uses the root index (tarred; see the module docstring caveat before using at scale)."""
294
+ if subset == "sample":
295
+ p = hf_hub_download(REPO_ID, "sample/index.parquet", repo_type="dataset")
296
+ df = pd.read_parquet(p)
297
+ # sample/ paths are relative to sample/; make them repo-relative
298
+ for col in ("video_path", "parquet_path"):
299
+ if col in df.columns:
300
+ df[col] = df[col].apply(lambda x: x if x.startswith("sample/") else f"sample/{x}")
301
+ return df
302
+ p = hf_hub_download(REPO_ID, "index.parquet", repo_type="dataset")
303
+ return pd.read_parquet(p)
304
+
305
+
306
+ def create_cs2_loader(
307
+ *,
308
+ subset: str = "sample",
309
+ index: pd.DataFrame | None = None,
310
+ maps: list[str] | None = None,
311
+ clip_len: int = 16,
312
+ target_fps: int = 16,
313
+ n_players: int = 4,
314
+ batch_size: int = 4,
315
+ num_workers: int = 0,
316
+ frame_size: tuple[int, int] | None = None,
317
+ valid_keys: list[str] | None = None,
318
+ shuffle: bool = True,
319
+ infinite: bool = True,
320
+ seed: int = 2025,
321
+ prefetch_factor: int = 2,
322
+ ) -> DataLoader:
323
+ """Drop-in replacement for MIRA's `create_loader`, streaming CS2-10k from the Hub (no disk).
324
+
325
+ Yields `(VideoActionBatch, list[ClipMeta])`. `n_players` perspectives per (match, round) are
326
+ grouped contiguously so the collate stacks them into one multi-perspective block, exactly as
327
+ MIRA's own loader does.
328
+ """
329
+ if index is None:
330
+ index = load_cs2_index(subset)
331
+ if maps:
332
+ index = index[index["map"].isin(maps)]
333
+ vocab = KeyVocab(tuple(valid_keys) if valid_keys else CS2_KEYS, on_unknown="ignore")
334
+ ds = Cs2StreamingDataset(
335
+ index,
336
+ vocab=vocab,
337
+ clip_len=clip_len,
338
+ target_fps=target_fps,
339
+ n_players=n_players,
340
+ frame_size=frame_size,
341
+ shuffle=shuffle,
342
+ infinite=infinite,
343
+ seed=seed,
344
+ from_tar=(subset == "full"), # full split = ranged tar-member fetch; sample = individual files
345
+ )
346
+ return DataLoader(
347
+ ds,
348
+ batch_size=batch_size * n_players, # n_players contiguous rows per sample-block
349
+ num_workers=num_workers,
350
+ collate_fn=_collate,
351
+ drop_last=True,
352
+ prefetch_factor=prefetch_factor if num_workers else None,
353
+ )
world_model_030000.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:059bb8f777295197ad46f14ccf81b0d26bedcd76bddaa991dd7dd17beb1a0ee3
3
+ size 50158667