| --- |
| language: en |
| license: [mit] |
| tags: |
| - text-classification |
| - ai-generated-text-detection |
| - modernbert |
| base_model: answerdotai/ModernBERT-large |
| pipeline_tag: text-classification |
| --- |
| |
|
|
| <div align="center"> |
|
|
| <img src="assets/vanguard_logo.svg" alt="Vanguard logo" width="128"/> |
| <h1> Vanguard — AI-Generated Text Detector</h1> |
| </div> |
|
|
|
|
| This model is a fine-tuned version of [ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) for binary classification of human-written vs. AI-generated text. It outputs a single probability, P(AI), indicating the likelihood that a given input was generated by a language model. |
| Vanguard AI-text detector was part of the [2nd-place system at PAN-CLEF 2026's Voight-Kampff Generative AI Detection leaderboard.](https://pan.webis.de/clef26/pan26-web/generated-content-analysis.html#leaderboard) |
|
|
| ## Model Details |
|
|
| - **Base model:** ModernBERT-large |
| - **Architecture:** ModernBERT-large with a single classification head (binary, sigmoid output) |
| - **Output:** A single scalar P(AI) in [0, 1]; a decision threshold of 0.5 is used by default, where P(AI) > 0.5 indicates AI-generated text |
| - **Language:** English |
| - **License:** MIT License |
|
|
| ## Training Data |
|
|
| This model was trained on approximately 1.1 million texts from three datasets: DACTYL 2.0, LLMTrace, and MAGA-Bench. |
|
|
| ## Evaluation |
|
|
| The model was evaluated against two leading open-source AI text detectors, [Fakespot](https://huggingface.co/PLACEHOLDER) and [Desklib](https://huggingface.co/PLACEHOLDER), across ten benchmark datasets. Three of these (dactyl-v2.0, llm-trace-eng, maga) are in-distribution with respect to this model's training data; the remaining seven (beemo, coconuts, detectrl, dolly-cosmopedia, originalityai, realdet, uchicago) are out-of-distribution and were not seen during training. |
|
|
| All F1 scores are macro-averaged and computed at a decision threshold of P(AI) = 0.5. |
|
|
| ### Results |
|
|
| | Dataset | AUROC | Macro-F1 | |
| |---|---|---| |
| | dactyl-v2.0 † | 0.9903 | 0.9709 | |
| | llm-trace-eng † | 0.9909 | 0.9661 | |
| | maga † | 0.9992 | 0.9901 | |
| | beemo | 0.8560 | 0.6973 | |
| | coconuts | 0.9792 | 0.8690 | |
| | detectrl | 0.9370 | 0.8849 | |
| | dolly-cosmopedia | 0.9958 | 0.8884 | |
| | originalityai | 0.9322 | 0.7776 | |
| | realdet | 0.9540 | 0.9177 | |
| | uchicago | 0.9784 | 0.9105 | |
|
|
| † In-distribution (training data overlap) |
|
|
| **Out-of-distribution averages:** AUROC 0.9475, Macro-F1 0.8493 |
|
|
| ### Comparison to Baselines (OOD average) |
|
|
| | Model | AUROC | Macro-F1 | |
| |---|---|---| |
| | Fakespot | 0.9315 | 0.8029 | |
| | Desklib | 0.9213 | 0.7837 | |
| | **Vanguard** | **0.9475** | **0.8493** | |
|
|
|
|
|
|
| <img src="assets/modernbert_auroc_by_dataset.svg" width="1000"> |
| <img src="assets/modernbert_f1_by_dataset.svg" width="1000"> |
|
|
|
|
| ### Notes on Evaluation |
|
|
| - Fakespot and Desklib use a two-head softmax architecture and were evaluated using their native argmax decision rule, which is mathematically equivalent to thresholding P(AI) at 0.5. |
| - This model outperforms both baselines on most out-of-distribution datasets, with the exception of coconuts (Desklib) and originalityai (Fakespot), where the baselines hold an edge. |
| - In-distribution performance is substantially higher than out-of-distribution performance, which is expected and should be taken into account when interpreting the headline averages; OOD results are more representative of expected real-world generalization. |
|
|
| ### Limitations and Out-of-Scope Use |
|
|
| This model should not be used as the sole basis for high-stakes decisions such as academic penalties or employment actions, given the false positive/negative rates documented below. Performance also degrades on text distributions not represented in training data; see evaluation results. |
|
|
|
|
| ## Citation |
|
|
| ``` |
| @article{thorat2026panclef, |
| title={Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection}, |
| author={Thorat, Shantanu}, |
| journal={Working Notes of CLEF}, |
| year={2026} |
| } |
| ``` |
|
|
