README.md

---
language:
  - en
license: apache-2.0
library_name: transformers
tags:
  - binary-analysis
  - file-type-detection
  - byte-level
  - classification
  - mime-type
  - roformer
  - rope
  - security
pipeline_tag: text-classification
base_model: magic-bert-50m-roformer-mlm
model-index:
  - name: magic-bert-50m-roformer-classification
    results:
      - task:
          type: text-classification
          name: File Type Classification
        metrics:
          - name: Probing Accuracy
            type: accuracy
            value: 93.7
          - name: Silhouette Score
            type: silhouette
            value: 0.663
          - name: F1 (Weighted)
            type: f1
            value: 0.933
---

# Magic-BERT 50M RoFormer Classification

A RoFormer-based transformer model fine-tuned for binary file type classification. This model achieves 93.7% classification accuracy across 106 MIME types, making it the **recommended choice for production file type detection**.

## Why Not Just Use libmagic?

For intact files starting at byte 0, libmagic works well. But libmagic matches *signatures at fixed offsets*. Magic-BERT learns *structural patterns* throughout the file, enabling use cases where you don't have clean file boundaries:

- **Network streams**: Classifying packet payloads mid-connection, before headers arrive
- **Disk forensics**: Identifying file types during carving, when scanning raw disk images without filesystem metadata
- **Fragment analysis**: Working with partial files, slack space, or corrupted data
- **Adversarial contexts**: Detecting file types when magic bytes are stripped, spoofed, or deliberately misleading

## Model Description

This model extends magic-bert-50m-roformer-mlm with contrastive learning fine-tuning. It uses Rotary Position Embeddings (RoPE) and produces highly discriminative embeddings for file type classification.

| Property | Value |
|----------|-------|
| Parameters | 42.0M (+ 0.45M classifier head) |
| Hidden Size | 512 |
| Projection Dimension | 256 |
| Number of Classes | 106 MIME types |
| Base Model | magic-bert-50m-roformer-mlm |
| Position Encoding | RoPE (Rotary Position Embeddings) |

### Tokenizer

The tokenizer uses the Binary BPE methodology introduced in [Bommarito (2025)](https://arxiv.org/abs/2511.17573). The original Binary BPE tokenizers (available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k)) were trained exclusively on executable binaries (ELF, PE, Mach-O). This tokenizer uses the same BPE training approach but was trained on a diverse corpus spanning 106 file types.

## Intended Uses

**Primary use cases:**
- Production file type classification
- MIME type detection from binary content
- Embedding-based file similarity search
- Security analysis and content filtering

This is the recommended model for file classification tasks due to its combination of high accuracy (93.7%) and parameter efficiency (42M parameters).

## Detailed Use Cases

### Network Traffic Analysis
When inspecting packet payloads, you often see file data mid-stream—TCP reassembly may give you bytes 1500-3000 of a PDF before you ever see byte 0. Traditional signature matching fails here. Classification embeddings can identify file types from interior content.

### Disk Forensics & File Carving
During disk image analysis, you scan raw bytes looking for file boundaries. Tools like Scalpel rely on header/footer signatures, but many files lack clear footers. This model can score byte ranges for file type probability, helping identify carved fragments or validate carving results.

### Incident Response
Malware often strips or modifies magic bytes to evade detection. Polyglot files (valid as multiple types) exploit signature-based tools. Learning structural patterns provides a second opinion that doesn't rely solely on the first few bytes.

### Similarity Search
The embedding space (256-dimensional, L2-normalized) enables similarity search across file collections: "find files structurally similar to this sample" for malware clustering, duplicate detection, or content-based retrieval.

## Architecture: RoPE vs Absolute Position Embeddings

This model uses **Rotary Position Embeddings (RoPE)**, which encode position through rotation matrices in attention. This differs from the Magic-BERT variant which uses absolute position embeddings.

| Metric | RoFormer (this) | Magic-BERT |
|--------|-----------------|------------|
| Classification Accuracy | **93.7%** | 89.7% |
| Silhouette Score | **0.663** | 0.55 |
| F1 (Weighted) | **0.933** | 0.886 |
| Parameters | **42.5M** | 59M |
| Fill-mask Retention | 14.5% | **41.8%** |

This model achieves higher classification accuracy with fewer parameters, making it the preferred choice for production deployment when only classification is needed.

## MLM vs Classification: Two-Phase Training

This is the **Phase 2 (Classification)** model built on RoFormer. The training pipeline has two phases:

| Phase | Model | Task | Purpose |
|-------|-------|------|---------|
| Phase 1 | magic-bert-50m-roformer-mlm | Masked Language Modeling | Learn byte-level patterns and file structure |
| **Phase 2** | **This model** | Contrastive Learning | Optimize embeddings for file type discrimination |

### Two-Phase Training

| Phase | Steps | Learning Rate | Objective |
|-------|-------|---------------|-----------|
| 1: MLM Pre-training | 100,000 | 1e-4 | Masked Language Modeling |
| 2: Contrastive Fine-tuning | 50,000 | 1e-6 | Supervised Contrastive Loss |

**Phase 2 specifics:**
- Frozen: Embeddings + first 4 transformer layers
- Learning rate: 100x lower than Phase 1
- Result: Significantly improved embedding quality for classification

## Evaluation Results

### Classification Performance

| Metric | Value |
|--------|-------|
| Linear Probe Accuracy | **93.7%** |
| F1 (Macro) | 0.829 |
| F1 (Weighted) | 0.933 |

### Embedding Quality

| Metric | Value |
|--------|-------|
| Silhouette Score | **0.663** |
| Separation Ratio | 4.00 |
| Intra-class Distance | 7.24 |
| Inter-class Distance | 28.98 |

The silhouette score of 0.663 indicates well-separated clusters, suitable for embedding-based retrieval and similarity search.

### Phase 1 → Phase 2 Improvement

| Metric | Phase 1 | Phase 2 | Change |
|--------|---------|---------|--------|
| Probing Accuracy | 85.0% | 93.7% | +8.7% |
| Silhouette Score | 0.328 | 0.663 | +102% |
| Separation Ratio | 2.65 | 4.00 | +51% |

## Supported MIME Types (106 Classes)

The model classifies files into 106 MIME types across these categories:

| Category | Count | Examples | Typical Accuracy |
|----------|-------|----------|------------------|
| application/ | 41 | PDF, ZIP, GZIP, Office docs, executables | >90% |
| text/ | 24 | Python, C, Java, HTML, XML, shell scripts | >80% |
| image/ | 18 | PNG, JPEG, GIF, WebP, TIFF, PSD | >95% |
| video/ | 9 | MP4, WebM, MKV, AVI, MOV | >90% |
| audio/ | 8 | MP3, FLAC, WAV, OGG, M4A | >90% |
| font/ | 3 | SFNT, WOFF, WOFF2 | >85% |
| other | 3 | biosig/atf, inode/x-empty, message/rfc822 | varies |

<details>
<summary>Click to expand full MIME type list</summary>

**application/** (41 types):
- application/SIMH-tape-data, application/encrypted, application/gzip
- application/javascript, application/json, application/msword
- application/mxf, application/octet-stream, application/pdf
- application/pgp-keys, application/postscript
- application/vnd.microsoft.portable-executable, application/vnd.ms-excel
- application/vnd.ms-opentype, application/vnd.ms-powerpoint
- application/vnd.oasis.opendocument.spreadsheet
- application/vnd.openxmlformats-officedocument.* (3 variants)
- application/vnd.rn-realmedia, application/vnd.wordperfect
- application/wasm, application/x-7z-compressed, application/x-archive
- application/x-bzip2, application/x-coff, application/x-dbf
- application/x-dosexec, application/x-executable
- application/x-gettext-translation, application/x-ms-ne-executable
- application/x-ndjson, application/x-object, application/x-ole-storage
- application/x-sharedlib, application/x-shockwave-flash
- application/x-tar, application/x-wine-extension-ini
- application/zip, application/zlib, application/zstd

**text/** (24 types):
- text/csv, text/html, text/plain, text/rtf, text/troff
- text/x-Algol68, text/x-asm, text/x-c, text/x-c++
- text/x-diff, text/x-file, text/x-fortran, text/x-java
- text/x-m4, text/x-makefile, text/x-msdos-batch, text/x-perl
- text/x-php, text/x-po, text/x-ruby, text/x-script.python
- text/x-shellscript, text/x-tex, text/xml

**image/** (18 types):
- image/bmp, image/fits, image/gif, image/heif, image/jpeg
- image/png, image/svg+xml, image/tiff, image/vnd.adobe.photoshop
- image/vnd.microsoft.icon, image/webp, image/x-eps, image/x-exr
- image/x-jp2-codestream, image/x-portable-bitmap
- image/x-portable-greymap, image/x-tga, image/x-xpixmap

**video/** (9 types):
- video/3gpp, video/mp4, video/mpeg, video/quicktime, video/webm
- video/x-ivf, video/x-matroska, video/x-ms-asf, video/x-msvideo

**audio/** (8 types):
- audio/amr, audio/flac, audio/mpeg, audio/ogg, audio/x-ape
- audio/x-hx-aac-adts, audio/x-m4a, audio/x-wav

**font/** (3 types):
- font/sfnt, font/woff, font/woff2

**other** (3 types):
- biosig/atf, inode/x-empty, message/rfc822

</details>

## How to Use

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

model = AutoModelForSequenceClassification.from_pretrained(
    "mjbommar/magic-bert-50m-roformer-classification", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("mjbommar/magic-bert-50m-roformer-classification")

model.eval()

# Classify a file
with open("example.pdf", "rb") as f:
    data = f.read(512)

# Decode bytes to string using latin-1 (preserves all byte values 0-255)
text = data.decode("latin-1")
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)
    predicted_id = outputs.logits.argmax(-1).item()
    confidence = torch.softmax(outputs.logits, dim=-1).max().item()

print(f"Predicted class: {predicted_id}")
print(f"Confidence: {confidence:.2%}")
```

### Embedding-Based Similarity Search

```python
# Get normalized embeddings (256-dim, L2-normalized)
with torch.no_grad():
    embeddings = model.get_embeddings(inputs["input_ids"], inputs["attention_mask"])
    # embeddings shape: [batch_size, 256]

# Compute cosine similarity
similarity = torch.mm(embeddings1, embeddings2.T)
```

### Loading MIME Type Labels

```python
from huggingface_hub import hf_hub_download
import json

mime_path = hf_hub_download("mjbommar/magic-bert-50m-roformer-classification", "mime_type_mapping.json")
with open(mime_path) as f:
    id_to_mime = {int(k): v for k, v in json.load(f).items()}

print(f"Predicted MIME type: {id_to_mime[predicted_id]}")
```

## Limitations

1. **MLM capability sacrificed:** Fill-mask accuracy drops to 14.5% after classification fine-tuning. Use the MLM variant if byte prediction is needed.

2. **Position bias:** Still present (~46% accuracy drop at offset 1000), though less relevant for classification than for fill-mask tasks.

3. **Ambiguous formats:** ZIP-based formats (DOCX, XLSX, JAR, APK) share similar structure and may be confused.

4. **Rare types:** Lower accuracy on underrepresented file types in training data.

## Model Selection Guide

| Use Case | Recommended Model | Reason |
|----------|-------------------|--------|
| **Production classification** | **This model** | Highest accuracy (93.7%), efficient (42M params) |
| Classification + fill-mask | magic-bert-50m-classification | Retains 41.8% fill-mask capability |
| Fill-mask / byte prediction | magic-bert-50m-roformer-mlm | Optimized for MLM |
| Research baseline | magic-bert-50m-mlm | Best perplexity (1.05) |

## Related Models

- **[magic-bert-50m-roformer-mlm](https://huggingface.co/mjbommar/magic-bert-50m-roformer-mlm)**: Base model before classification fine-tuning
- **[magic-bert-50m-mlm](https://huggingface.co/mjbommar/magic-bert-50m-mlm)**: Absolute position embedding variant (MLM)
- **[magic-bert-50m-classification](https://huggingface.co/mjbommar/magic-bert-50m-classification)**: Magic-BERT variant that retains better fill-mask capability (89.7% accuracy)

## Related Work

This model builds on the Binary BPE tokenization approach:

- **Binary BPE Paper**: [Bommarito (2025)](https://arxiv.org/abs/2511.17573) introduced byte-level BPE tokenization for binary analysis, demonstrating 2-3x compression over raw bytes for executable content.
- **Binary BPE Tokenizers**: Pre-trained tokenizers for executables are available at [mjbommar/binary-tokenizer-001-64k](https://huggingface.co/mjbommar/binary-tokenizer-001-64k).

**Key difference**: The original Binary BPE work focused on executable binaries (ELF, PE, Mach-O). Magic-BERT extends this to general file type understanding across 106 diverse formats, using a tokenizer trained on the broader dataset.

## Citation

A paper describing Magic-BERT, the training methodology, and the dataset is forthcoming.

```bibtex
@article{bommarito2025binarybpe,
  title={Binary BPE: A Family of Cross-Platform Tokenizers for Binary Analysis},
  author={Bommarito, Michael J., II},
  journal={arXiv preprint arXiv:2511.17573},
  year={2025}
}
```