Update README.md
Browse files
README.md
CHANGED
@@ -18,22 +18,23 @@ Here are the specifications of the process used:
|
|
18 |
- The MG-Verilog dataset was downloaded from Gatech-EIC/MG-Verilog. I have copied this dataset to my GitHub repository used to train the model.
|
19 |
- The dataset and the training script was uploaded to my Google Drive for easy access on Colab.
|
20 |
- A 4-bit quantized version of the base model is loaded using a quantization configuration using BitsAndBytesConfig and AutoModelForCausalLM.from_pretrained. The follwing are the specifications of the quantization configuration:
|
21 |
-
|
22 |
load_in_4bit=True,
|
23 |
bnb_4bit_use_double_quant=True,
|
24 |
bnb_4bit_quant_type="nf4",
|
25 |
bnb_4bit_compute_dtype=torch.bfloat16
|
|
|
26 |
|
27 |
- The model is prepared for k-bit training using prepare_model_for_kbit_training.
|
28 |
- The PEFT configuartion is set and a PeftModel is created from the quantized model. Here are the specifications of the PEFT configuration:
|
29 |
-
|
30 |
r=64,
|
31 |
lora_alpha=16,
|
32 |
target_modules=["q_proj", "v_proj"],
|
33 |
lora_dropout=0.05,
|
34 |
bias="none",
|
35 |
task_type=TaskType.CAUSAL_LM
|
36 |
-
|
37 |
- The Data Collator class implemented in the training script has been taken directly from qlora.py from the MG-Verilog GitHub (https://github.com/GATECH-EIC/mg-verilog)
|
38 |
|
39 |
## Citations
|
|
|
18 |
- The MG-Verilog dataset was downloaded from Gatech-EIC/MG-Verilog. I have copied this dataset to my GitHub repository used to train the model.
|
19 |
- The dataset and the training script was uploaded to my Google Drive for easy access on Colab.
|
20 |
- A 4-bit quantized version of the base model is loaded using a quantization configuration using BitsAndBytesConfig and AutoModelForCausalLM.from_pretrained. The follwing are the specifications of the quantization configuration:
|
21 |
+
```
|
22 |
load_in_4bit=True,
|
23 |
bnb_4bit_use_double_quant=True,
|
24 |
bnb_4bit_quant_type="nf4",
|
25 |
bnb_4bit_compute_dtype=torch.bfloat16
|
26 |
+
```
|
27 |
|
28 |
- The model is prepared for k-bit training using prepare_model_for_kbit_training.
|
29 |
- The PEFT configuartion is set and a PeftModel is created from the quantized model. Here are the specifications of the PEFT configuration:
|
30 |
+
```
|
31 |
r=64,
|
32 |
lora_alpha=16,
|
33 |
target_modules=["q_proj", "v_proj"],
|
34 |
lora_dropout=0.05,
|
35 |
bias="none",
|
36 |
task_type=TaskType.CAUSAL_LM
|
37 |
+
```
|
38 |
- The Data Collator class implemented in the training script has been taken directly from qlora.py from the MG-Verilog GitHub (https://github.com/GATECH-EIC/mg-verilog)
|
39 |
|
40 |
## Citations
|