Update README.md

README.md

@@ -13,126 +13,225 @@ This is a [sentence-transformers](https://www.SBERT.net) model: It maps sentence
 
 <!--- Describe your model here -->
 
-## Usage (Sentence-Transformers)
+---
+license: apache-2.0
+language:
+- en
+library_name: sentence-transformers
+tags:
+- bert
+- scibert
+- sentence-transformers
+- sentence-similarity
+- scientific-text
+- mirror
+- r-compatible
+base_model: pritamdeka/S-Scibert-snli-multinli-stsb
+pipeline_tag: sentence-similarity
+---
 
-Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
+# S-SciBERT (snli-multinli-stsb) — safetensors mirror for use from R
 
-```
-pip install -U sentence-transformers
-```
+This is a format-converted mirror of [`pritamdeka/S-Scibert-snli-multinli-stsb`](https://huggingface.co/pritamdeka/S-Scibert-snli-multinli-stsb), maintained for teaching a course on transformer-based topic modeling in R.
 
-Then you can use the model like this:
+The model itself is unchanged: same architecture, same weights, same tokenizer, same outputs as the upstream original. What's different is the on-disk format and provenance, both of which matter for a teaching context.
 
-```python
-from sentence_transformers import SentenceTransformer
-sentences = ["This is an example sentence", "Each sentence is converted"]
+## Why this mirror exists
 
-model = SentenceTransformer('pritamdeka/S-Scibert-snli-multinli-stsb')
-embeddings = model.encode(sentences)
-print(embeddings)
-```
+The upstream repo ships `pytorch_model.bin` (PyTorch pickle format) and no `tokenizer.json`. For Python users this works fine, but for R users working through the `torch` (libtorch) and `safetensors` R packages there is a more serious problem than just format inconvenience:
 
+**The upstream pickle file was saved while the model was on a CUDA device.** PyTorch's pickle format records the device of every tensor, so loading on a CPU-only machine fails with an `aten::empty_strided ... CUDA backend` error. Python's `torch.load(map_location='cpu')` rescues you from this, but R-torch's loader doesn't expose that argument, so the upstream file is effectively unusable from R unless you have a CUDA GPU available.
 
+This mirror adds:
 
-## Usage (HuggingFace Transformers)
-Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
+- `model.safetensors` — the same weights in [safetensors](https://huggingface.co/docs/safetensors) format. Safetensors files do not record device information at all, so they load cleanly regardless of where the model was originally saved or what hardware the user has.
 
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
+The fix is structural, not just cosmetic: safetensors solves a class of cross-device portability problems that pickle cannot, on top of being safer and faster to read.
 
+## What it is, briefly
 
-#Mean Pooling - Take attention mask into account for correct averaging
-def mean_pooling(model_output, attention_mask):
-    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
-    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
-    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+S-SciBERT is [SciBERT-cased](https://huggingface.co/allenai/scibert_scivocab_cased) fine-tuned for sentence similarity using the [sentence-transformers](https://www.sbert.net) framework. The fine-tuning data was the standard general-English similarity benchmark suite: SNLI (natural language inference), MultiNLI (multi-genre NLI), and STS-B (semantic textual similarity).
 
+The result is a model that combines SciBERT's scientific vocabulary (gene names, chemical terms, ML jargon) with sentence-transformer-quality embeddings — meaning the mean-pooled output vectors actually cluster well, which base SciBERT's do not.
 
-# Sentences we want sentence embeddings for
-sentences = ['This is an example sentence', 'Each sentence is converted']
+| Property | Value |
+|----------|-------|
+| Architecture | BERT-base + mean-pooling head |
+| Parameters | ~110M |
+| Embedding dimension | 768 |
+| Layers | 12 |
+| Attention heads | 12 |
+| Vocabulary | SciBERT scientific (cased, ~31K tokens) |
+| Pooling | Mean over tokens (masked by attention) |
+| Fine-tuning data | SNLI + MultiNLI + STS-B |
+| Training max_seq_length | 75 tokens |
+| Case sensitivity | Cased |
 
-# Load model from HuggingFace Hub
-tokenizer = AutoTokenizer.from_pretrained('pritamdeka/S-Scibert-snli-multinli-stsb')
-model = AutoModel.from_pretrained('pritamdeka/S-Scibert-snli-multinli-stsb')
+## When to use this model
 
-# Tokenize sentences
-encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
+**Good fit:**
+- Topic modeling, clustering, or semantic search over scientific text (papers, abstracts, scientific tweets, GitHub issues from research codebases).
+- Domains where SciBERT's vocabulary is an advantage: biomedical, computer science, computational biology, machine learning, chemistry.
+- Sentence-level or paragraph-level inputs.
 
-# Compute token embeddings
-with torch.no_grad():
-    model_output = model(**encoded_input)
+**Less good fit:**
+- General web text — a general-purpose sentence-transformer like `all-MiniLM-L6-v2` or `all-mpnet-base-v2` will likely match or beat S-SciBERT on non-scientific content.
+- Document-level inputs (full papers): the model was fine-tuned on sequences of 75 tokens. It still runs on longer inputs (up to the BERT-base ceiling of 512), but quality degrades for content past the trained length. For long documents, split into sentences or paragraphs and embed those individually.
+- Languages other than English: the fine-tuning data is English-only.
+
+## Usage from R
 
-# Perform pooling. In this case, max pooling.
-sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
+This mirror works with a pure-R BERT inference pipeline built on top of the `torch` (libtorch) R package, with no Python at runtime:
 
-print("Sentence embeddings:")
-print(sentence_embeddings)
+```r
+source("bert_r.R")
+enc <- load_hf_bert("NetworkIsLife/S-SciBert_DAFS")
+
+emb <- embed_texts(enc$model, enc$tokenizer,
+                   c("CRISPR-Cas9 enables targeted gene editing.",
+                     "Glioblastoma exhibits invasive growth patterns.",
+                     "Gradient descent minimizes a loss function."),
+                   max_length = 128)
+dim(emb)   # 3 x 768
+
+# Cosine similarity (embeddings are L2-normalized by default)
+sims <- emb %*% t(emb)
+round(sims, 3)
+# Rows 1 and 2 should be more similar to each other (both biomedical)
+# than either is to row 3 (machine learning)
 ```
 
+For topic modeling with the full pipeline:
 
+```r
+source("bertopic_r.R")
+fit <- fit_bertopic(enc, docs = my_abstracts,
+                    umap_n_neighbors = 15,
+                    hdbscan_min_pts  = 10)
+print_topics(fit)
+```
 
-## Evaluation Results
+For long-term reproducibility in course materials, pin to a specific revision:
 
-<!--- Describe how your model was evaluated -->
+```r
+enc <- load_hf_bert(
+  "NetworkIsLife/S-SciBert_DAFS",
+  weights_path = hfhub::hub_download(
+    "NetworkIsLife/S-SciBert_DAFS",
+    "model.safetensors",
+    revision = "MAIN_COMMIT_HASH_HERE"
+  )
+)
+```
 
-For an automated evaluation of this model, see the *Sentence Embeddings Benchmark*: [https://seb.sbert.net](https://seb.sbert.net?model_name={MODEL_NAME})
+Replace `MAIN_COMMIT_HASH_HERE` with the commit hash visible in this repo's commit history.
 
+## Usage from Python
 
-## Training
-The model was trained with the parameters:
+Either of the standard idioms works:
 
-**DataLoader**:
+```python
+# Via sentence-transformers (easiest)
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer("NetworkIsLife/S-SciBert_DAFS")
+embeddings = model.encode([
+    "CRISPR-Cas9 enables targeted gene editing.",
+    "Glioblastoma exhibits invasive growth patterns."
+])
 
-`torch.utils.data.dataloader.DataLoader` of length 90 with parameters:
-```
-{'batch_size': 64, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}
+# Via transformers (with manual mean pooling)
+from transformers import AutoTokenizer, AutoModel
+import torch
+import torch.nn.functional as F
+
+tok = AutoTokenizer.from_pretrained("NetworkIsLife/S-SciBert_DAFS")
+mod = AutoModel.from_pretrained("NetworkIsLife/S-SciBert_DAFS").eval()
+
+enc = tok(sentences, padding=True, truncation=True, return_tensors="pt", max_length=128)
+with torch.no_grad():
+    out = mod(**enc).last_hidden_state
+    m = enc["attention_mask"].unsqueeze(-1).float()
+    pooled = (out * m).sum(1) / m.sum(1).clamp(min=1e-9)
+    embeddings = F.normalize(pooled, p=2, dim=1)
 ```
 
-**Loss**:
+## Files in this repo
 
-`sentence_transformers.losses.CosineSimilarityLoss.CosineSimilarityLoss` 
+| File | Source | Purpose |
+|------|--------|---------|
+| `model.safetensors` | converted from upstream `pytorch_model.bin` | model weights, modern format (device-agnostic) |
+| `pytorch_model.bin` | copied from upstream | model weights, legacy format (kept for compatibility) |
+| `config.json` | copied from upstream | BERT architecture parameters |
+| `vocab.txt` | copied from upstream | SciBERT WordPiece vocabulary |
+| `tokenizer_config.json` | copied from upstream (if present) | tokenizer settings (do_lower_case, special tokens) |
+| `README.md` | this file | provenance and usage |
 
-Parameters of the fit()-Method:
-```
-{
-    "callback": null,
-    "epochs": 4,
-    "evaluation_steps": 1000,
-    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
-    "max_grad_norm": 1,
-    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
-    "optimizer_params": {
-        "lr": 2e-05
-    },
-    "scheduler": "WarmupLinear",
-    "steps_per_epoch": null,
-    "warmup_steps": 36,
-    "weight_decay": 0.01
-}
-```
+## Provenance and verification
 
+The `model.safetensors` file in this repo was produced by HuggingFace's official `SFconvertbot` (the same automated conversion used across thousands of HuggingFace repos). The conversion is purely a re-serialization — every tensor in the safetensors file is bit-identical to the corresponding tensor in `pytorch_model.bin`. No re-training, no quantization, no precision loss.
 
-## Full Model Architecture
-```
-SentenceTransformer(
-  (0): Transformer({'max_seq_length': 75, 'do_lower_case': False}) with Transformer model: BertModel 
-  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
-)
+You can verify this yourself in Python:
+
+```python
+import torch
+from safetensors.torch import load_file
+
+# map_location='cpu' is needed because the upstream pickle was saved on GPU
+a = torch.load("pytorch_model.bin", map_location="cpu", weights_only=True)
+b = load_file("model.safetensors")
+assert set(a.keys()) == set(b.keys())
+for k in a:
+    assert torch.equal(a[k].cpu(), b[k]), f"Mismatch in {k}"
+print("Bit-identical.")
 ```
 
-## Citing & Authors
+End-to-end verification (cosine similarities computed via the upstream model and via this mirror should agree to ~1e-6):
 
-<!--- Describe where people can find more information -->
+```python
+from sentence_transformers import SentenceTransformer
+import numpy as np
+
+sentences = [
+    "CRISPR-Cas9 enables targeted gene editing.",
+    "Glioblastoma exhibits invasive growth.",
+    "Gradient descent minimizes a loss function."
+]
+a = SentenceTransformer("pritamdeka/S-Scibert-snli-multinli-stsb").encode(sentences)
+b = SentenceTransformer("NetworkIsLife/S-SciBert_DAFS").encode(sentences)
+print("max |Δ| =", np.abs(a - b).max())   # should be ~1e-6 or smaller
+```
 
-If you use the model kindly cite the following work
+## License and citation
 
-```
+This mirror inherits the upstream license: **Apache 2.0**. If you use this model in academic work, please cite the original paper:
+
+```bibtex
 @inproceedings{deka2021unsupervised,
-  title={Unsupervised Keyword Combination Query Generation from Online Health Related Content for Evidence-Based Fact Checking},
+  title={Unsupervised Keyword Combination Query Generation from
+         Online Health Related Content for Evidence-Based Fact Checking},
   author={Deka, Pritam and Jurek-Loughrey, Anna},
-  booktitle={The 23rd International Conference on Information Integration and Web Intelligence},
+  booktitle={The 23rd International Conference on Information Integration
+             and Web-based Applications & Services},
   pages={267--277},
   year={2021}
 }
-```
+```
+
+And consider also citing the underlying SciBERT paper that this model fine-tunes from:
+
+```bibtex
+@inproceedings{beltagy-etal-2019-scibert,
+  title = "{SciBERT}: A Pretrained Language Model for Scientific Text",
+  author = "Beltagy, Iz and Lo, Kyle and Cohan, Arman",
+  booktitle = "Proceedings of EMNLP-IJCNLP",
+  year = "2019",
+  url = "https://www.aclweb.org/anthology/D19-1371"
+}
+```
+
+Original model: [`pritamdeka/S-Scibert-snli-multinli-stsb`](https://huggingface.co/pritamdeka/S-Scibert-snli-multinli-stsb) by Pritam Deka.
+Base model: [`allenai/scibert_scivocab_cased`](https://huggingface.co/allenai/scibert_scivocab_cased) by the Allen Institute for AI.
+
+## Maintenance
+
+This is a teaching artifact for a course on transformer-based topic modeling in R. It will not be updated except to fix conversion errors. For the canonical, maintained version of S-SciBERT, see the [upstream repo](https://huggingface.co/pritamdeka/S-Scibert-snli-multinli-stsb).