See axolotl config

axolotl version: 0.4.1

adapter: lora
base_model: trl-internal-testing/tiny-random-LlamaForCausalLM
bf16: true
chat_template: llama3
dataset_prepared_path: null
datasets:
- data_files:
  - 57fd039527663fc0_train_data.json
  ds_type: json
  format: custom
  path: /workspace/input_data/57fd039527663fc0_train_data.json
  type:
    field_input: knowledge
    field_instruction: intent
    field_output: response
    format: '{instruction} {input}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
debug: null
device_map:
  ? ''
  : 0,1,2,3,4,5,6,7
early_stopping_patience: 2
eval_max_new_tokens: 128
eval_steps: 100
eval_table_size: null
flash_attention: true
gradient_accumulation_steps: 8
gradient_checkpointing: true
group_by_length: false
hub_model_id: Alphatao/a35f6753-dab1-417f-bb87-722cc70dc198
hub_repo: null
hub_strategy: null
hub_token: null
learning_rate: 0.0002
load_best_model_at_end: true
load_in_4bit: false
load_in_8bit: false
local_rank: null
logging_steps: 1
lora_alpha: 128
lora_dropout: 0.1
lora_fan_in_fan_out: null
lora_model_dir: null
lora_r: 64
lora_target_linear: true
lora_target_modules:
- q_proj
- k_proj
- v_proj
- o_proj
lr_scheduler: cosine
max_grad_norm: 1.0
max_steps: 21483
micro_batch_size: 4
mlflow_experiment_name: /tmp/57fd039527663fc0_train_data.json
model_type: AutoModelForCausalLM
num_epochs: 2
optimizer: adamw_bnb_8bit
output_dir: miner_id_24
pad_to_sequence_len: true
resume_from_checkpoint: null
s2_attention: null
sample_packing: false
save_steps: 100
sequence_len: 2048
strict: false
tf32: true
tokenizer_type: AutoTokenizer
train_on_inputs: false
trust_remote_code: true
val_set_size: 0.03351206434316354
wandb_entity: null
wandb_mode: online
wandb_name: 02fa1c1f-9647-4e4f-8cb2-27ce7de9079c
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: 02fa1c1f-9647-4e4f-8cb2-27ce7de9079c
warmup_steps: 10
weight_decay: 0.0
xformers_attention: null

a35f6753-dab1-417f-bb87-722cc70dc198

This model is a fine-tuned version of trl-internal-testing/tiny-random-LlamaForCausalLM on the None dataset. It achieves the following results on the evaluation set:

• Loss: 10.3098

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 10
• training_steps: 9013

Training results

Training Loss	Epoch	Step	Validation Loss
10.3777	0.0002	1	10.3780
10.3459	0.0222	100	10.3433
10.3325	0.0444	200	10.3329
10.3343	0.0666	300	10.3281
10.3273	0.0888	400	10.3251
10.3251	0.1110	500	10.3227
10.3264	0.1331	600	10.3210
10.3221	0.1553	700	10.3199
10.325	0.1775	800	10.3187
10.3212	0.1997	900	10.3177
10.324	0.2219	1000	10.3171
10.3246	0.2441	1100	10.3163
10.3238	0.2663	1200	10.3159
10.3265	0.2885	1300	10.3153
10.3205	0.3107	1400	10.3147
10.3155	0.3329	1500	10.3144
10.3144	0.3551	1600	10.3138
10.3184	0.3773	1700	10.3134
10.3207	0.3994	1800	10.3133
10.3117	0.4216	1900	10.3129
10.3193	0.4438	2000	10.3127
10.3116	0.4660	2100	10.3124
10.3208	0.4882	2200	10.3122
10.3151	0.5104	2300	10.3122
10.3176	0.5326	2400	10.3116
10.3191	0.5548	2500	10.3117
10.3198	0.5770	2600	10.3116
10.319	0.5992	2700	10.3113
10.3187	0.6214	2800	10.3114
10.3158	0.6436	2900	10.3108
10.3246	0.6657	3000	10.3108
10.3208	0.6879	3100	10.3113
10.3156	0.7101	3200	10.3105
10.316	0.7323	3300	10.3104
10.3226	0.7545	3400	10.3103
10.3189	0.7767	3500	10.3102
10.3199	0.7989	3600	10.3101
10.3232	0.8211	3700	10.3101
10.316	0.8433	3800	10.3099
10.3195	0.8655	3900	10.3100
10.3209	0.8877	4000	10.3097
10.3124	0.9098	4100	10.3100
10.3115	0.9320	4200	10.3098

Framework versions

• PEFT 0.13.2
• Transformers 4.46.0
• Pytorch 2.5.0+cu124
• Datasets 3.0.1
• Tokenizers 0.20.1