File size: 2,847 Bytes
1ffffb4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 |
---
language: en
license: apache-2.0
tags:
- intellite
- "small-language-model"
- transformers
- pytorch
- conversational-ai
- huggingface
- low-resource
---
# IntellIte Chat
IntellIte Chat is a lightweight conversational AI model (~45M parameters) designed for warm, engaging dialogue and basic reasoning. Part of the *IntellIte* series, it delivers efficient performance on modest hardware, complete with streaming data loading, episodic memory buffers, and RAG-based knowledge augmentation.
---
## ⚙️ Key Features
- **Small & Efficient**: ~45M parameters, ideal for edge devices and academic projects.
- **Streaming Data**: Uses Hugging Face `IterableDataset` for on-the-fly data without local storage constraints.
- **Memory Buffer**: Maintains the last 200 messages for coherent multi-turn conversations.
- **RAG Integration**: FAISS-based retrieval for up-to-date knowledge augmentation.
- **Content Safety**: Built-in filters to enforce conversational guidelines.
- **Extensible API**: Hook into `generate_with_plugins()` for custom prompts or downstream tasks.
---
## 💾 Installation
```bash
pip install transformers datasets faiss-cpu torch huggingface-hub
```
---
## 🚀 Quick Start
```python
from il import generate_with_plugins
response = generate_with_plugins(
prompt="Hello, how's it going?",
source="wiki",
k=3,
max_new_tokens=100
)
print(response)
```
---
## 🛠️ Training Pipeline
Run the main training script:
```bash
export HF_TOKEN=<your_hf_token>
python il.py --hf_token $HF_TOKEN --seed 42
```
The script will:
1. Stream Wikipedia, CodeParrot, and grade-school math datasets.
2. Apply cosine LR scheduling, weight decay, and label smoothing.
3. Run simple evals (2 chat, 1 code prompt) at each epoch end.
4. Save & push the best model to `ProCreations/IntellIte` on Hugging Face.
---
## 📊 Evaluation & Monitoring
A `SimpleEvalCallback` runs designated chat/code prompts each epoch, logging outputs for quick sanity checks.
---
## 🔧 Configuration Options
Edit `il.py` to customize:
- **Batch Sizes, LR, Scheduler** via `TrainingArguments`.
- **Retrieval Sources**: adjust `k` and index sources.
- **Memory Buffer**: change size or filter rules.
---
## 🌱 Fine‑Tuning on Custom Data
1. Prepare your dataset as a Hugging Face `Dataset` or `IterableDataset`.
2. Interleave with base streams and pass to the Trainer.
3. Use `--resume_from_checkpoint` to continue an interrupted run.
---
## 🤝 Contributing
Contributions welcome! Steps:
1. Fork the repo.
2. Create a feature branch.
3. Submit a PR with clear descriptions and tests.
---
## 📜 License
This project is licensed under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).
---
❤️ Developed by ProCreations under the *IntellIte* brand.
|