Model card: JevBench v1.4 public 0.892, Decision Index 0.2.1 52.89, vs base, GIFs

.gitattributes

@@ -34,3 +34,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/2048.gif filter=lfs diff=lfs merge=lfs -text
+assets/snake.gif filter=lfs diff=lfs merge=lfs -text
+assets/tetris.gif filter=lfs diff=lfs merge=lfs -text
+assets/vs_base.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,322 @@
+---
+license: cc-by-nc-4.0
+base_model: Qwen/Qwen3.5-35B-A3B
+library_name: transformers
+pipeline_tag: image-text-to-text
+tags: [decision-model, calibration, typed-decisions, classification, multimodal, moe]
+---
+
+# JPT-35B-A3B
+
+[![License](https://img.shields.io/badge/license-CC_BY--NC_4.0-lightgrey)](#❯❯-license)
+[![JevBench v1.4 public](https://img.shields.io/badge/JevBench_v1.4_public-0.892-E8663D)](#❯-jevbench-v140)
+[![Decision Index 0.2.1](https://img.shields.io/badge/Decision_Index_0.2.1-52.89-E8663D)](#❯-decision-index-021)
+[![llm2jev](https://img.shields.io/badge/serve_with-llm2jev-blue)](https://github.com/tic-top/llm2jev)
+
+[JPT-0.8B](https://huggingface.co/kirp/jpt-0.8b) · [JPT-4B](https://huggingface.co/kirp/jpt-4b) · [JPT-9B](https://huggingface.co/kirp/jpt-9b) · **JPT-35B-A3B** · [llm2jev](https://github.com/tic-top/llm2jev)
+
+| 2048 | Snake | Tetris |
+|:-:|:-:|:-:|
+| ![2048](assets/2048.gif) | ![Snake](assets/snake.gif) | ![Tetris](assets/tetris.gif) |
+
+<sub>Each move is one `choice` question answered in one forward pass (30 moves each, seed 0, SGLang, recorded headless).</sub>
+
+## What is JPT-35B-A3B
+
+**JPT-35B-A3B is the largest JPT: an open decision model that takes a situation and typed questions and returns a
+calibrated probability for every option from one forward pass.** No generated explanation, no reasoning tokens —
+latency is one prefill, and only 3B of its 35B parameters are active per token.
+
+It implements the typed-decision interface introduced by [Jev](https://typesafe.ai) from TypeSafe AI [1]: a caller
+sends a state plus questions, and each question is one of three types. JPT is an independent model, not derived from
+Jev and not trained on Jev outputs; it is an open alternative behind the same interface.
+
+| Question type | What it answers | Options |
+|---|---|---|
+| `choice` | pick one | 2–255 labels |
+| `score` | a level on an ordered scale | the scale's levels |
+| `noul` | yes / no | `true`, `false` |
+
+Built on [Qwen/Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B), a mixture-of-experts model (3B active
+parameters per token): a LoRA fine-tune merged into full weights, with the same recipe and data as JPT-4B and JPT-9B.
+The vision tower is unchanged, so screenshots, photos and video frames work next to the text state.
+
+> Probabilities use one temperature T = 1.087, fit once on a held-out split — never per benchmark.
+> Read them as well-ordered confidence; ECE varies by task (0.03–0.17).
+
+## ❯❯ Benchmarks
+
+### ❯ JevBench v1.4.0
+
+**[JevBench](https://github.com/fstandhartinger/jevbench) [2] scores general typed decisions; JPT-35B-A3B gets 206 of
+the 231 public items, above every system in the v1.4 results.** v1.4.0 has 231 public items (frozen since v1.2) and
+308 sealed items that only the maintainer can run, so JPT-35B-A3B has no official v1.4 score yet.
+
+![JevBench v1.4.0 public accuracy](assets/jevbench.png)
+
+<details>
+<summary>Table</summary>
+
+| System | Params | Public accuracy (231) | Sealed accuracy (308) | v1.4 score |
+|---|---:|---:|---:|---:|
+| **JPT-35B-A3B** | 35B-A3B | **0.892** | pending | pending |
+| JPT-4B (ours) | 4B | 0.879 | pending | pending |
+| Jev 1.13.0 (TypeSafe AI, API) | closed | 0.866 | 0.367 | 63.3 |
+| JPT-9B (ours) | 9B | 0.857 | pending | pending |
+| Winnow-12B Q8 | 12B | 0.857 | 0.331 | 55.6 |
+| JevK5 v0.2.0 | 27B | 0.853 | 0.331 | 62.0 |
+| decider-35b-a3b | 35B-A3B | 0.831 | 0.315 | 41.2 |
+| Hopper | — | 0.823 | 0.341 | 59.4 |
+| Qwen3.5-35B-A3B, same prompt, zero-shot (our run) | 35B-A3B | 0.844 | — | — |
+
+By tier: easy 48/48, standard 71/72, hard 87/111. Latency on our runner (tensor parallel 4 on A100 40 GB, no other
+load on that replica): p50 56 ms, p95 103 ms.
+
+Source: other rows from `results/v1.4/jevbench-v1.4-results.json` at jevbench commit `2fa63fa` (2026-09-23).
+
+</details>
+
+### ❯ Decision Index 0.2.1
+
+**[Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index) [3] (0.2.1, 2026-09-27) is the
+broadest test: 38 scored benchmarks in five areas, chance-corrected. JPT-35B-A3B scores 52.89, 5.8 points above
+Decider 35B-A3B on the same base model and 6.0 above JPT-9B.** We ran the full frozen suite (150,759 requests, every
+one answered) through llm2jev over SGLang and submitted it as
+[apolinario/decision-index#7](https://github.com/apolinario/decision-index/pull/7); it is not on the live board until
+the maintainer validates it.
+
+![Decision Index 0.2.1](assets/decision_index.png)
+
+<details>
+<summary>Table</summary>
+
+| Model | Params | Decision Index 0.2.1 |
+|---|---:|---:|
+| Jev 1.13.0 (TypeSafe AI, API) | closed | 57.91 |
+| Surogate Rune 26B-A4B v3 | 26B-A4B | 57.44 |
+| Decider chat · Gemma-4-31B | 31B | 57.33 |
+| AutoJev-27B | 27B | 56.40 |
+| simple-jev · Qwen3.8-27B | 27B | 55.74 |
+| Jebadiah 27B | 27B | 54.67 |
+| Eikos-27B | 27B | 53.13 |
+| **JPT-35B-A3B** | 35B-A3B | **52.89** |
+| reflex 27B | 27B | 52.16 |
+| Decider chat · Qwen3.6-27B | 27B | 51.35 |
+| Winnow-12B | 12B | 50.02 |
+| Decider 35B-A3B | 35B-A3B | 47.11 |
+| JPT-9B (ours) | 9B | 46.89 |
+
+</details>
+
+Source: full run and `scores.json` in [kirp/decision-index-results-jpt-35b-a3b](https://huggingface.co/datasets/kirp/decision-index-results-jpt-35b-a3b)
+(gated: it carries the suite's GPQA/HLE item text). Other rows: live board `data/index-v0.2.1.json` (generated 2026-09-27 16:59 UTC).
+
+**By area, against Jev 1.13.0 on the same items and scorer:** JPT-35B-A3B is level with Jev on language and
+retrieval, behind it on knowledge, tools and arts, and ahead of it on 7 of the 38 index benchmarks.
+
+![Decision Index 0.2.1 by area](assets/di_areas.png)
+
+<details>
+<summary>Per-area and per-benchmark skill vs Jev 1.13.0</summary>
+
+| Area | Jev 1.13.0 | JPT-35B-A3B |
+|---|---:|---:|
+| Knowledge | 51.4 | 37.0 |
+| Language | 62.0 | 61.9 |
+| Retrieval | 55.4 | **55.7** |
+| Tools | 75.1 | 69.5 |
+| Arts | 37.7 | 34.8 |
+
+| Area | Benchmark | Jev 1.13.0 | JPT-35B-A3B |
+|---|---|---:|---:|
+| Arts | BPoMP | 81.8 | 80.4 |
+| Arts | ForecastBench | 30.6 | 25.9 |
+| Arts | Habermas Machine | 21.5 | 18.2 |
+| Arts | Humicroedit | 23.7 | **24.7** |
+| Arts | New Yorker | 62.6 | 60.5 |
+| Arts | POP909-CL | 15.9 | 11.0 |
+| Arts | cfcolor | 28.8 | 24.6 |
+| Games | ChessBench | 9.8 | 8.9 |
+| Knowledge | BBH | 89.7 | 60.8 |
+| Knowledge | CLadder | 45.3 | 38.0 |
+| Knowledge | CRUXEval | 57.1 | 48.8 |
+| Knowledge | GPQA Diamond | 71.4 | 32.0 |
+| Knowledge | GSM8K | 75.6 | 67.5 |
+| Knowledge | HLE | 4.7 | 0.0 |
+| Knowledge | MMLU-Pro | 80.5 | 56.8 |
+| Knowledge | MuSR | 46.1 | 35.3 |
+| Knowledge | SATA-Bench | 25.4 | 21.6 |
+| Language | ACOS | 27.3 | 18.8 |
+| Language | ANLI | 62.2 | 57.2 |
+| Language | ContractNLI | 59.1 | **73.4** |
+| Language | FinEntity | 80.8 | **90.3** |
+| Language | HellaSwag | 92.7 | 89.0 |
+| Language | NLI4CT | 69.0 | 66.0 |
+| Language | RAGTruth | 51.3 | 49.1 |
+| Language | VAST | 46.9 | **52.8** |
+| Language | WinoGrande | 83.9 | 67.3 |
+| Language | iSarcasmEval | 36.3 | **49.3** |
+| Retrieval | Amazon ESCI | 43.8 | 41.5 |
+| Retrieval | BANKING77 | 79.5 | 77.2 |
+| Retrieval | BRIGHT | 40.6 | 37.2 |
+| Retrieval | CLINC150+OOS | 89.2 | 85.0 |
+| Retrieval | HoVer | 45.7 | 35.1 |
+| Retrieval | PhishNChips phishing decisions | 25.1 | **51.5** |
+| Tools | API-Bank | 88.0 | 85.4 |
+| Tools | BFCL | 94.3 | **94.8** |
+| Tools | Home appliance simulator | 52.3 | 35.2 |
+| Tools | ToolRet | 59.9 | 57.9 |
+| Tools | When2Call | 74.6 | 65.9 |
+
+Chance-corrected skill × 100 (0 = random, 100 = perfect). Jev's numbers are its official entry on the live board
+(`jev-1.13.0`); ours are from the same kit and suite.
+
+</details>
+
+### ❯ Against its base model
+
+**Same prompt, same serving path, Qwen3.5-35B-A3B zero-shot vs JPT-35B-A3B: fine-tuning lifts every row but ANLI r1, where the strong base is level, and buys +15 points on typed decisions at the cost of calibration there (ECE 0.031 → 0.171).**
+
+![JPT-35B-A3B vs Qwen3.5-35B-A3B](assets/vs_base.png)
+
+<details>
+<summary>Table, with calibration</summary>
+
+| Benchmark (version, n) | What it tests | JPT-35B-A3B | Qwen3.5-35B-A3B zero-shot | Jev 1.13.0 |
+|---|---|---:|---:|---:|
+| JevBench v1.4.0 public hard tier [2] (111) | hardest general decisions | **0.784** (ECE 0.066, Brier 0.297) | 0.703 (ECE 0.069, Brier 0.390) | — |
+| Typed decisions test (ours, 2,000) | in-distribution typed decisions | **0.803** (ECE 0.171, Brier 0.323) | 0.651 (ECE 0.031, Brier 0.449) | — |
+| ANLI r1 / r2 / r3 [4] (dev) | adversarial NLI | 0.773 / **0.680 / 0.737** | **0.777** / 0.650 / 0.703 | — |
+| Banking77 [5] / MASSIVE 1.1 [6] (en / de / zh) | intent classification | **0.777 / 0.893 / 0.867 / 0.833** | 0.720 / 0.840 / 0.787 / 0.773 | — |
+| AG News [7] / Emotion [8] / SST-5 [9] (test) | out-of-distribution classification | **0.917 / 0.593 / 0.593** | 0.887 / 0.517 / 0.543 | — |
+| ScreenSpot-v2 [11] / Screen2Words [12] / ERQA [13] (images) | GUI grounding, screen summary, embodied reasoning | — | — | — |
+| EnvBench v0.1 (ours) public / held-out [10] (skill 0–100) | sequential decisions in game envs | — | — | — |
+
+Banking77, MASSIVE and typed rows are in-distribution (train splits in the mix, test items not). The image evals and
+EnvBench have not been run on this checkpoint yet.
+
+Jev 1.13.0 has no official score on these splits (our own test/dev cuts, EnvBench, and the image sets), so its column is "—"; its official scores on the Decision Index versions of ANLI and BANKING77 are in the per-benchmark table above. Running the Jev API on these splits would fill them.
+
+</details>
+
+## ❯❯ Quick Start
+
+**Two pieces: an engine that holds the weights, and [llm2jev](https://pypi.org/project/llm2jev/) (>= 0.6.1) in
+front of it, reading option probabilities off the engine.** The bf16 weights are 66 GB, so below 80 GB per GPU use
+tensor parallelism.
+
+### ⚡ SGLang (recommended)
+
+```bash
+python -m sglang.launch_server --model-path kirp/jpt-35b-a3b --port 30000 --tp 4 --mem-fraction-static 0.7 \
+  --context-length 32768 --mamba-scheduler-strategy extra_buffer &  # Qwen3.5's DeltaNet layers need this flag
+llm2jev --model kirp/jpt-35b-a3b --backend sglang --url http://127.0.0.1:30000 --port 8080 --temperature 1.087
+```
+
+Tested with SGLang 0.5.18 on 4 × A100 40 GB. `--mem-fraction-static 0.7` leaves room for the vision tower's warm-up;
+at the 0.8 we use for smaller sizes, the warm-up runs out of memory at TP 4.
+
+### 🔁 vLLM
+
+```bash
+vllm serve kirp/jpt-35b-a3b --tensor-parallel-size 4 --max-logprobs 256 --return-tokens-as-token-ids \
+  --enable-scale-out --port 8000
+llm2jev --model kirp/jpt-35b-a3b --backend vllm --url http://127.0.0.1:8000 --port 8080 --temperature 1.087
+```
+
+The three flags after `--tensor-parallel-size` are required: without them every request is a bare HTTP 400. Not yet
+tested with this checkpoint.
+
+### 📨 Ask it a question
+
+```python
+import requests
+r = requests.post("http://127.0.0.1:8080/v1/systemone", json={
+    "state": "Refund policy: full refund within 30 days of purchase; 50% until day 60; none after.\n"
+             "Order 1182 was bought on 3 March and returned on 20 April.",
+    "questions": {
+        "refund": {"type": "choice", "instructions": "What refund does order 1182 get?",
+                   "criteria": {"full": "Full refund", "half": "50% refund", "none": "No refund"}},
+        "late":   {"type": "noul", "instructions": "Was the return made after day 30?",
+                   "criteria": {"true": "Yes", "false": "No"}}}})
+print(r.json()["answers"])   # each answer has the per-option probabilities
+```
+
+### 🖼️ With images
+
+```python
+state = [{"role": "user", "content": [
+    {"type": "image", "image": "https://example.com/screen.png"},
+    {"type": "text", "text": "Task: open the settings page. Numbered boxes mark clickable elements."}]}]
+questions = {"click": {"type": "choice", "instructions": "Which box should be clicked?",
+                       "criteria": {"1": None, "2": None, "3": None, "4": None, "5": None}}}
+```
+
+## ❯❯ Training
+
+**LoRA with a Brier loss on 49,221 typed questions, one epoch, trained with Megatron-SWIFT and merged into full
+weights.**
+
+| Part | What it is |
+|---|---|
+| Method | LoRA r=16, alpha 32 on every attention, DeltaNet, shared-expert and routed-expert projection of the language model; router and vision tower untouched |
+| Loss | multi-class Brier over the option labels, on llm2jev's chat prompt with thinking disabled |
+| Data | 49,221 questions in 32,835 records; one epoch over two option-shuffled copies (94,172 rows) |
+| Schedule | lr 5e-5, cosine decay, 5% warm-up, global batch 64, 1,471 steps; final validation Brier loss 0.209 |
+| Hardware | Megatron-SWIFT on 8 × A100 40 GB: expert parallelism 8, sequence parallelism, 2 h 17 min |
+| Held out | no item from JevBench, EnvBench held-out seeds, the Decision Index frozen suite or our typed test split; 8-gram overlap check vs JevBench |
+
+<details>
+<summary>Data sources</summary>
+
+The same mix as JPT-4B and JPT-9B (`mix_train_env_v11`):
+
+- public classification, NLI, QA, preference and safety datasets;
+- long legal and contract documents (ContractNLI, MAUD, LegalBench, ConditionalQA, ShARC);
+- table and numeric reasoning (TAT-QA, MultiHiertt);
+- multi-hop QA (MuSiQue, BEIR);
+- agent and tool traces (Mind2Web, AgentTraj, ToolACE);
+- community typed-decision sets;
+- oracle-labelled rollouts from 20 small game and puzzle environments;
+- programmatically generated rule-arithmetic items (dates, time zones, day counts, caps; labels computed by code);
+- 1,289 long policy / contract / regulation documents (4,471 questions) written by an LLM (GPT-6 Luna) with no
+  JevBench item shown to it.
+
+</details>
+
+## ❯❯ Limitations
+
+- **Arithmetic and dates** are its weakest area: it answers from the evidence given and has no reasoning phase by design.
+- **Overconfident in-distribution.** At the global temperature, typed-decisions test ECE is 0.171: confidence on
+  in-distribution typed questions runs above accuracy.
+- **Up to 255 options** are accepted; training covered up to 77 (Banking77), so beyond that quality is not established.
+- **English first.** Other languages come only from a few multilingual classification sets.
+- **Images are untested** on this checkpoint; they go zero-shot through the base vision tower.
+- **Heavy to host.** 66 GB of bf16 weights need several GPUs, even though only 3B parameters are active per token.
+
+## ❯❯ References
+
+1. TypeSafe AI. *Jev.* https://typesafe.ai
+2. F. Standhartinger. *JevBench*, v1.4.0. https://github.com/fstandhartinger/jevbench
+3. *Decision Index*, edition 0.2.1. https://huggingface.co/spaces/multimodalart/jev-decision-index
+4. Nie et al. *Adversarial NLI.* ACL 2020.
+5. Casanueva et al. *Efficient Intent Detection with Dual Sentence Encoders* (Banking77). NLP4ConvAI 2020.
+6. FitzGerald et al. *MASSIVE.* ACL 2023.
+7. Zhang et al. *Character-level Convolutional Networks for Text Classification* (AG News). NeurIPS 2015.
+8. Saravia et al. *CARER: Contextualized Affect Representations for Emotion Recognition.* EMNLP 2018.
+9. Socher et al. *Recursive Deep Models for Semantic Compositionality* (SST). EMNLP 2013.
+10. *EnvBench*, v0.1 (ours, frozen 2026-09-23; not yet public): programmatically solved game, planning and rule decisions with exact gold answers.
+11. Wu et al. *OS-Atlas* (ScreenSpot-v2). 2024.
+12. Wang et al. *Screen2Words.* UIST 2021.
+13. Gemini Robotics Team. *Gemini Robotics* (ERQA). 2025.
+
+## ❯❯ License
+
+**CC BY-NC 4.0.** The weights derive from Qwen3.5-35B-A3B (Apache-2.0), but some training datasets allow only
+non-commercial or research use, so the model is released for non-commercial use.
+
+---
+
+<small>JPT-35B-A3B is an independent open model that implements a typed-decision interface (noul, choice and score
+questions answered with probabilities). It is not affiliated with, endorsed by or derived from TypeSafe AI or its Jev
+model, and it was not trained on Jev outputs.</small>