|
|
--- |
|
|
language: |
|
|
- en |
|
|
license: mit |
|
|
datasets: |
|
|
- Trelis/tiny-shakespeare |
|
|
pipeline_tag: text-generation |
|
|
library_name: transformers |
|
|
--- |
|
|
# Decoder Language Model |
|
|
Ein kleiner autoregressiver Decoder-only Transformer, trainiert auf Tiny Shakespeare. |
|
|
|
|
|
## Architektur |
|
|
- d_model=128, num_layers=2, nhead=4 |
|
|
- ~500k Parameter |
|
|
|
|
|
## Metriken |
|
|
- Loss (Train): 0.6342 |
|
|
- Perplexity (Train): 1.8854 |
|
|
|
|
|
## Laden |
|
|
```python |
|
|
from transformers import GPT2Tokenizer |
|
|
import torch |
|
|
from model import DecoderLanguageModel |
|
|
|
|
|
tokenizer = GPT2Tokenizer.from_pretrained("ahmadisakina/decoder-language-model") |
|
|
model = DecoderLanguageModel(vocab_size=tokenizer.vocab_size, d_model=128, nhead=4, num_layers=2) |
|
|
model.load_state_dict(torch.load("pytorch_model.bin")) |
|
|
model.eval() |
|
|
``` |
