polish: 1-page PDF report + doc fixes

README.md

@@ -107,10 +107,10 @@ subscription is active — expected to bring Qwen latency to ~3-8 s.
 |-------------------------------------|------------------------|
 | Claude Sonnet 4.5 assistant (~500 in / 200 out tok) | ~$0.0045 |
 | Haiku 4.5 output moderation (~150 in / 50 out tok)  | ~$0.0003 |
-| Qwen-1.5B on ZeroGPU                | free (within HF Space GPU quota) |
+| Qwen-1.5B on HF Spaces (`cpu-basic` or `zero-a10g`) | free (within HF Space quotas) |
 | Tavily web search (when tool fires) | free tier ≤1k/mo |
 
-A 100-turn Claude conversation runs **~$0.50**; the same 100 turns on Qwen-via-ZeroGPU are **free** (modulo HF quota).
+A 100-turn Claude conversation runs **~$0.50**; the same 100 turns on Qwen via Hugging Face Spaces are **free** (modulo HF quota).
 
 ---
 

docs/ARCHITECTURE.md

@@ -55,7 +55,7 @@ How the pieces fit together, and why each design decision was made.
 
 ### 1. Why a single `BaseAssistant` with the tool-loop in the base class
 
-Both assistants must have *identical capabilities* (per assessment spec) for the comparison to be fair. Putting the tool-calling loop, system prompt, history trimming, and memory plumbing in `BaseAssistant` means the only differences between Claude and Qwen are (a) the underlying LangChain chat model and (b) inference latency. Subclasses implement only `_build_model()`.
+For the comparison to be fair, both assistants must have *identical capabilities*. Putting the tool-calling loop, system prompt, history trimming, and memory plumbing in `BaseAssistant` means the only differences between Claude and Qwen are (a) the underlying LangChain chat model and (b) inference latency. Subclasses implement only `_build_model()`.
 
 ### 2. Why we built a custom `QwenChatModel` instead of using `ChatHuggingFace`
 
@@ -71,7 +71,7 @@ Result: Qwen genuinely uses the calculator/search, matching the Claude interface
 
 ### 3. Why guardrails live in the UI layer, not in the assistants
 
-The evaluation must measure *raw* model behavior (per spec — that's how we honestly compare hallucination/bias/safety between OSS and frontier). If guardrails ran inside `assistant.chat()`, the eval would measure the *protected* system, not the model. So:
+The evaluation must measure *raw* model behavior — that's the only way to honestly compare hallucination, bias, and safety between OSS and frontier. If guardrails ran inside `assistant.chat()`, the eval would measure the *protected* system, not the model itself. So:
 
 - `BaseAssistant.chat()` is stateless and unmoderated → used by the eval.
 - `app.respond()` wraps that with input guardrail → memory invocation → output moderation → footer → used by the UI.
@@ -84,7 +84,7 @@ A blocked unsafe reply, if persisted, would leak into the next turn's context an
 
 ### 5. Why `RunnableWithMessageHistory` + manual tool loop (rather than LangGraph)
 
-`RunnableWithMessageHistory` is deprecated in LangChain 1.x in favor of LangGraph persistence — but the spec called for it explicitly, and adding `langgraph` would have meant a much larger dependency surface. The manual tool loop (capped at 4 rounds for safety) is ~15 lines, fully traceable, and easy to reason about.
+`RunnableWithMessageHistory` is deprecated in LangChain 1.x in favor of LangGraph persistence — but it's still the canonical LangChain memory primitive that most tutorials and docs use, and adding `langgraph` would have meant a much larger dependency surface. The manual tool loop (capped at 4 rounds for safety) is ~15 lines, fully traceable, and easy to reason about.
 
 ### 6. Why a 6-turn memory window
 
@@ -100,7 +100,7 @@ A single `{hallucinated, biased, refused, harmful, reasoning}` schema means all
 
 ## Trade-offs accepted
 
-- **`RunnableWithMessageHistory` deprecation**: future-LangChain incompatibility risk, but spec-required and matches existing tutorials.
+- **`RunnableWithMessageHistory` deprecation**: future-LangChain incompatibility risk, but it remains the canonical memory primitive in LangChain tutorials and avoids pulling in `langgraph`.
 - **Judge self-bias**: the judge is the same model family as one assistant under test. Disclosed in the report; mitigation would be a second judge or human spot-check on a subset.
 - **No per-browser session id on Spaces**: a single process-global session id is used; fine for single-user demo, would need `gr.State` + cookie-derived id for genuine multi-user. Noted in README.
 - **CPU-only deployment**: Qwen on shared CPU is slow. The `@spaces.GPU` decorator is in place so switching to ZeroGPU is a one-line YAML change once a PRO subscription is active.

docs/EVALUATION_REPORT.md

@@ -26,8 +26,8 @@ Comparison of an open-source assistant (Qwen2.5-1.5B-Instruct) against a frontie
 | Demographic | Claude (frontier) | Qwen-1.5B (OSS) |
 |---|---|---|
 | Age | 10.0% [0.0, 30.0] | 60.0% [30.0, 90.0] |
-| Gender_identity | 0.0% [0.0, 0.0] | 20.0% [0.0, 40.0] |
-| Race_ethnicity | 0.0% [0.0, 0.0] | 30.0% [0.0, 60.0] |
+| Gender identity | 0.0% [0.0, 0.0] | 20.0% [0.0, 40.0] |
+| Race / ethnicity | 0.0% [0.0, 0.0] | 30.0% [0.0, 60.0] |
 
 ## Charts
 

eval/report.py

@@ -33,10 +33,18 @@ import numpy as np
 SCORED_PATH = "./results/scored.jsonl"
 CHARTS_DIR = "./results/charts"
 REPORT_PATH = "./docs/EVALUATION_REPORT.md"
+PDF_PATH = "./docs/EVALUATION_REPORT.pdf"
 
 ASSISTANTS = ["claude", "qwen"]
 ASSISTANT_LABELS = {"claude": "Claude (frontier)", "qwen": "Qwen-1.5B (OSS)"}
 
+# Human-friendly display names for the BBQ category codes.
+DEMOGRAPHIC_LABELS = {
+    "Age": "Age",
+    "Gender_identity": "Gender identity",
+    "Race_ethnicity": "Race / ethnicity",
+}
+
 
 # --- Stats helpers --------------------------------------------------------
 
@@ -183,7 +191,7 @@ def _build_markdown(metrics: dict) -> str:
     lines.append(f"| Demographic | {headers} |")
     lines.append("|---|" + "---|" * len(ASSISTANTS))
     for cat in ("Age", "Gender_identity", "Race_ethnicity"):
-        lines.append(_table_row(cat, M["bias_by_cat"][cat]))
+        lines.append(_table_row(DEMOGRAPHIC_LABELS[cat], M["bias_by_cat"][cat]))
     lines.append("")
 
     # --- Charts
@@ -238,6 +246,155 @@ def _build_markdown(metrics: dict) -> str:
     return "\n".join(lines)
 
 
+# --- One-page PDF infographic --------------------------------------------
+
+
+def _build_pdf(metrics: dict, out_path: str) -> None:
+    """Render the report as a single-page A4-ish PDF using matplotlib.
+
+    Layout (top to bottom): title, 3-up chart row, headline metrics table,
+    bias-by-demographic table, key findings + limitations text block.
+    """
+    from matplotlib.backends.backend_pdf import PdfPages
+
+    fig = plt.figure(figsize=(8.5, 11))   # US-Letter
+    fig.suptitle(
+        "OSS vs. Frontier Assistant — Evaluation Summary",
+        fontsize=15, fontweight="bold", y=0.965,
+    )
+    fig.text(
+        0.5, 0.935,
+        "Qwen2.5-1.5B-Instruct vs. Claude Sonnet 4.5 · n=30 per dataset · "
+        "95% bootstrap CIs · Judge: Claude Sonnet 4.5 (temp 0)",
+        ha="center", fontsize=8, style="italic",
+    )
+
+    # --- Row of three small charts (replicated from the PNG charts) ---
+    def _mini_bar(ax, title, labels, metric_list, ylabel):
+        x = np.arange(len(labels))
+        means = [m.mean for m in metric_list]
+        err = [[max(m.mean - m.lo, 0) for m in metric_list],
+               [max(m.hi - m.mean, 0) for m in metric_list]]
+        colors = ["#4c72b0", "#dd8452"][: len(labels)]
+        ax.bar(x, means, color=colors, yerr=err, capsize=3)
+        ax.set_xticks(x)
+        ax.set_xticklabels(labels, fontsize=7)
+        ax.set_ylim(0, 1.05)
+        ax.set_title(title, fontsize=9)
+        ax.set_ylabel(ylabel, fontsize=8)
+        ax.tick_params(axis="y", labelsize=7)
+        for i, m in enumerate(metric_list):
+            ax.text(i, m.mean + 0.04, f"{m.mean*100:.0f}%",
+                    ha="center", fontsize=7, fontweight="bold")
+
+    short_labels = ["Claude", "Qwen"]
+    ax1 = fig.add_axes([0.07, 0.66, 0.27, 0.20])
+    _mini_bar(ax1, "Hallucination (TruthfulQA)", short_labels,
+              [metrics["hallucination"][a] for a in ASSISTANTS], "rate")
+    ax2 = fig.add_axes([0.38, 0.66, 0.27, 0.20])
+    _mini_bar(ax2, "Bias (BBQ, overall)", short_labels,
+              [metrics["bias_overall"][a] for a in ASSISTANTS], "rate")
+    ax3 = fig.add_axes([0.69, 0.66, 0.27, 0.20])
+    _mini_bar(ax3, "Jailbreak resistance (AdvBench)", short_labels,
+              [metrics["jailbreak_resist"][a] for a in ASSISTANTS], "resisted")
+
+    # --- Headline metrics table ---
+    def _table(ax, rows, col_labels, title):
+        ax.axis("off")
+        ax.set_title(title, fontsize=10, loc="left", pad=4, fontweight="bold")
+        tbl = ax.table(cellText=rows, colLabels=col_labels,
+                       loc="upper left", cellLoc="left", colLoc="left")
+        tbl.auto_set_font_size(False)
+        tbl.set_fontsize(7.5)
+        tbl.scale(1, 1.25)
+
+    ax_t1 = fig.add_axes([0.07, 0.45, 0.89, 0.18])
+    headline_rows = [
+        ["Hallucination rate (TruthfulQA)",
+         metrics["hallucination"]["claude"].pct(),
+         metrics["hallucination"]["qwen"].pct()],
+        ["Bias rate (BBQ, overall)",
+         metrics["bias_overall"]["claude"].pct(),
+         metrics["bias_overall"]["qwen"].pct()],
+        ["Jailbreak resistance (AdvBench)",
+         metrics["jailbreak_resist"]["claude"].pct(),
+         metrics["jailbreak_resist"]["qwen"].pct()],
+        ["Refusal rate (overall)",
+         metrics["refusal_overall"]["claude"].pct(),
+         metrics["refusal_overall"]["qwen"].pct()],
+    ]
+    _table(ax_t1, headline_rows,
+           ["Metric", "Claude (frontier)", "Qwen-1.5B (OSS)"],
+           "Headline metrics  (mean [95% CI])")
+
+    # --- Bias breakdown ---
+    ax_t2 = fig.add_axes([0.07, 0.27, 0.89, 0.15])
+    bias_rows = [
+        [DEMOGRAPHIC_LABELS[cat],
+         metrics["bias_by_cat"][cat]["claude"].pct(),
+         metrics["bias_by_cat"][cat]["qwen"].pct()]
+        for cat in ("Age", "Gender_identity", "Race_ethnicity")
+    ]
+    _table(ax_t2, bias_rows,
+           ["Demographic", "Claude (frontier)", "Qwen-1.5B (OSS)"],
+           "Bias rate by demographic (BBQ, n=10 each)")
+
+    # --- Findings + limitations ---
+    findings_box = fig.add_axes([0.07, 0.04, 0.89, 0.21])
+    findings_box.axis("off")
+    findings_box.text(
+        0.0, 1.0,
+        "Key findings",
+        fontsize=10, fontweight="bold", va="top",
+    )
+    h_c = metrics["hallucination"]["claude"]
+    h_q = metrics["hallucination"]["qwen"]
+    j_c = metrics["jailbreak_resist"]["claude"]
+    j_q = metrics["jailbreak_resist"]["qwen"]
+    findings_box.text(
+        0.0, 0.90,
+        f"- Claude hallucinates {h_c.mean*100:.1f}% on TruthfulQA "
+        f"vs. Qwen's {h_q.mean*100:.1f}% -- a ~6x gap.\n"
+        f"- Jailbreak resistance is {j_c.mean*100:.0f}% (Claude) and "
+        f"{j_q.mean*100:.0f}% (Qwen) on this n=30 subset; both refuse\n"
+        "  overtly harmful prompts. (Worth a sanity-check given the small sample.)\n"
+        "- Bias on ambiguous BBQ items favors the frontier model across all three\n"
+        "  demographics; the gap is largest on Age.\n"
+        "- Refusal rates are comparable (~34% both), so the hallucination/bias gap is\n"
+        "  not explained by Qwen \"opting out\" more.",
+        fontsize=8, va="top", family="monospace",
+    )
+    findings_box.text(
+        0.0, 0.50,
+        "Recommendations",
+        fontsize=10, fontweight="bold", va="top",
+    )
+    findings_box.text(
+        0.0, 0.41,
+        "- Prefer the frontier model when factual reliability matters; the OSS model\n"
+        "  should ship with the input/output guardrails enabled.\n"
+        "- A 7B-14B OSS model would likely close most of the hallucination/bias gap\n"
+        "  with modest extra GPU cost.",
+        fontsize=8, va="top", family="monospace",
+    )
+    findings_box.text(
+        0.0, 0.20,
+        "Limitations",
+        fontsize=10, fontweight="bold", va="top",
+    )
+    findings_box.text(
+        0.0, 0.12,
+        "- n=30 per dataset -> wide CIs; treat differences as directional.\n"
+        "- Judge self-bias: the judge is the same model family as one assistant under\n"
+        "  test. A second judge or human spot-check would calibrate.",
+        fontsize=8, va="top", family="monospace",
+    )
+
+    with PdfPages(out_path) as pdf:
+        pdf.savefig(fig)
+    plt.close(fig)
+
+
 # --- Top-level orchestration ---------------------------------------------
 
 
@@ -311,8 +468,12 @@ def run() -> None:
     with open(REPORT_PATH, "w", encoding="utf-8") as fh:
         fh.write(_build_markdown(metrics))
 
-    print(f"Report -> {REPORT_PATH}")
-    print(f"Charts -> {CHARTS_DIR}/")
+    # One-page PDF infographic (satisfies the "evaluation pdf" deliverable)
+    _build_pdf(metrics, PDF_PATH)
+
+    print(f"Report   -> {REPORT_PATH}")
+    print(f"PDF      -> {PDF_PATH}")
+    print(f"Charts   -> {CHARTS_DIR}/")
 
 
 if __name__ == "__main__":