File size: 1,389 Bytes
640b720 1b3b5f5 640b720 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
---
base_model: meta-llama/meta-llama-3.1-8b-instruct
tags:
- llama adapter
- trl
- llama3.1 8b
license: apache-2.0
language:
- en
---
## Model Overview
A LoRA (Low-Rank Adaptation) decomposed from base model from base to instruct
## Model Details
- Base Model: (meta-llama/Llama-3.1-8B) - (meta-llama/Llama-3.1-8B-instruct)
- Adaptation Method: LoRA
## Training Configuration
### Training Hyperparameters
- Rank (r): 16 -> 16
- Alpha: 1 -> 16
### LoRA Configuration
- Rank (r): 16
- Alpha: 16
- Target Modules:
- `q_proj` (Query projection)
- `k_proj` (Key projection)
- `v_proj` (Value projection)
- `o_proj` (Output projection)
- `up_proj` (Upsampling projection)
- `down_proj` (Downsampling projection)
- `gate_proj` (Gate projection)
## Usage
This adapter must be used in conjunction with the base Llama-3.1-8B-instruct model.
### Loading the Model
```python
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load base model
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-instruct")
# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "path_to_adapter")
```
## Limitations and Biases
- This adapter might inherits some limitations and biases present in the base Llama-3.1-8B-instruct model |