|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- meta-llama/Llama-3.1-8B-instruct |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- lora |
|
|
- adapter |
|
|
- Math |
|
|
- CoT |
|
|
--- |
|
|
|
|
|
## Model Details |
|
|
- Base Model: meta-llama/Llama-3.1-8B-instruct + 340, 338, 344 adapters with dare_ties_svd |
|
|
- KTO 800st 2 epoch |
|
|
|
|
|
## Datasets: |
|
|
- revision and rewriting txt |
|
|
|
|
|
### Source Adapters |
|
|
All source adapters share the following configuration: |
|
|
- Rank (r): 32 |
|
|
- Alpha: 32 |
|
|
- Target Modules: |
|
|
- q_proj (Query projection) |
|
|
- k_proj (Key projection) |
|
|
- v_proj (Value projection) |
|
|
- o_proj (Output projection) |
|
|
- up_proj (Upsampling projection) |
|
|
- down_proj (Downsampling projection) |
|
|
- gate_proj (Gate projection) |
|
|
|
