dangpranay commited on
Commit
6f9bf9a
·
verified ·
1 Parent(s): 89f7195

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -3
README.md CHANGED
@@ -18,22 +18,23 @@ Here are the specifications of the process used:
18
  - The MG-Verilog dataset was downloaded from Gatech-EIC/MG-Verilog. I have copied this dataset to my GitHub repository used to train the model.
19
  - The dataset and the training script was uploaded to my Google Drive for easy access on Colab.
20
  - A 4-bit quantized version of the base model is loaded using a quantization configuration using BitsAndBytesConfig and AutoModelForCausalLM.from_pretrained. The follwing are the specifications of the quantization configuration:
21
-
22
  load_in_4bit=True,
23
  bnb_4bit_use_double_quant=True,
24
  bnb_4bit_quant_type="nf4",
25
  bnb_4bit_compute_dtype=torch.bfloat16
 
26
 
27
  - The model is prepared for k-bit training using prepare_model_for_kbit_training.
28
  - The PEFT configuartion is set and a PeftModel is created from the quantized model. Here are the specifications of the PEFT configuration:
29
-
30
  r=64,
31
  lora_alpha=16,
32
  target_modules=["q_proj", "v_proj"],
33
  lora_dropout=0.05,
34
  bias="none",
35
  task_type=TaskType.CAUSAL_LM
36
-
37
  - The Data Collator class implemented in the training script has been taken directly from qlora.py from the MG-Verilog GitHub (https://github.com/GATECH-EIC/mg-verilog)
38
 
39
  ## Citations
 
18
  - The MG-Verilog dataset was downloaded from Gatech-EIC/MG-Verilog. I have copied this dataset to my GitHub repository used to train the model.
19
  - The dataset and the training script was uploaded to my Google Drive for easy access on Colab.
20
  - A 4-bit quantized version of the base model is loaded using a quantization configuration using BitsAndBytesConfig and AutoModelForCausalLM.from_pretrained. The follwing are the specifications of the quantization configuration:
21
+ ```
22
  load_in_4bit=True,
23
  bnb_4bit_use_double_quant=True,
24
  bnb_4bit_quant_type="nf4",
25
  bnb_4bit_compute_dtype=torch.bfloat16
26
+ ```
27
 
28
  - The model is prepared for k-bit training using prepare_model_for_kbit_training.
29
  - The PEFT configuartion is set and a PeftModel is created from the quantized model. Here are the specifications of the PEFT configuration:
30
+ ```
31
  r=64,
32
  lora_alpha=16,
33
  target_modules=["q_proj", "v_proj"],
34
  lora_dropout=0.05,
35
  bias="none",
36
  task_type=TaskType.CAUSAL_LM
37
+ ```
38
  - The Data Collator class implemented in the training script has been taken directly from qlora.py from the MG-Verilog GitHub (https://github.com/GATECH-EIC/mg-verilog)
39
 
40
  ## Citations