Hugging Face's logo

UniversalAlgorithmic
/
SPG

TensorBoard
Safetensors
Model card Files Metrics Community
1
SPG
File size: 24,921 Bytes 
aa3ad5a
7920b56
72e75ab
f7f9ac2
7920b56
56c67bf
7920b56
 
56c67bf
 
 
 
83a2cac
6fe54e3
56c67bf
83a2cac
 
56c67bf
83a2cac
 
56c67bf
83a2cac
 
7920b56
 
 
83a2cac
56c67bf
 
 
83a2cac
 
 
 
 
 
 
 
 
 
 
 
7920b56
 
72e75ab
56c67bf
 
 
 
 
 
 
83a2cac
 
56c67bf
83a2cac
 
56c67bf
83a2cac
 
56c67bf
83a2cac
 
56c67bf
83a2cac
 
56c67bf
83a2cac
 
56c67bf
83a2cac
 
56c67bf
83a2cac
 
56c67bf
 
83a2cac
72e75ab
 
83a2cac
 
 
 
 
 
 
 
72e75ab
 
 
7920b56
 
 
 
 
72e75ab
7920b56
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
83a2cac
7920b56
 
 
 
 
 
 
 
 
83a2cac
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
 
# SPG: Sequential Policy Gradient: Lightweight Reinforcement Learning for Model Performance

> 🚀 If you're using Jupyter or Colab, you can follow the demo and run it on a single GPU:
[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/#fileId=https%3A//huggingface.co/UniversalAlgorithmic/SPG/blob/main/demo_nas.ipynb)

## Model Zoo: Adaptive Hyperparameter Optimization (HPO) via SPG Algorithm


`Table 1: Performance of pre-trained vs. SPG-retrained models on ImageNet-1K`
| Model | SPG  | # Params | Acc@1 (%) | Acc@5 (%) | Weights | Command to reproduce |
|-------|------|----------|-----------|-----------|---------|----------------------|
| MobileNet-V2 | ❌ | 3.5 M | 71.878 | 90.286 | <a href='https://download.pytorch.org/models/mobilenet_v2-b0353104.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#mobilenetv2'>Recipe</a> |
| MobileNet-V2 | ✅HPO | 3.5 M | 72.104 | 90.316 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/mobilenetv2/model_32.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
| MobileNet-V2 | ✅NAS | 3.5 M | 72.208 | 90.822 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/mobilenetv2/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> |
| ResNet-50 | ❌ | 25.6 M | 76.130 | 92.862 | <a href='https://download.pytorch.org/models/resnet50-0676ba61.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#resnet'>Recipe</a> |
| ResNet-50 | ✅HPO | 25.6 M | 77.234 | 93.322 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/resnet50/model_35.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
| ResNet-50 | ✅NAS | 25.6 M | 80.970 | 95.481 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> |
| EfficientNet-V2-M | ❌ | 54.1 M | 85.112 | 97.156 | <a href='https://download.pytorch.org/models/efficientnet_v2_m-dc08266a.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#efficientnet-v2'>Recipe</a> |
| EfficientNet-V2-M | ✅HPO | 54.1 M | 85.218 | 97.208 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/efficientnet_v2_m/model_7.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/efficientnet_v2_m-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
| EfficientNet-V2-M | ✅NAS | 54.1 M | 85.347 | 97.424 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/efficientnet_v2_m/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/efficientnet_v2_m-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> |
| ViT-B16 | ❌ | 86.6 M | 81.072 | 95.318 | <a href='https://download.pytorch.org/models/vit_b_16-c867db91.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#vit_b_16'>Recipe</a> |
| ViT-B16 | ✅HPO | 86.6 M | 81.092 | 95.304 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/vit_b_16/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/vit_b_16-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> |
| ViT-B16 | ✅NAS | 86.6 M | 81.114 | 95.320 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/vit_b_16/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/vit_b_16-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> |



`Table 2: Performance of pre-trained vs. SPG-retrained models. All models are evaluated a subset of COCO val2017, on the 21 categories that are present in the Pascal VOC dataset.`

| Model               | SPG | # Params | mIoU (%)   | pixelwise Acc (%)   | Weights | Command to reproduce |
|---------------------|-----|----------|------------|---------------------|---------|----------------------|
| FCN-ResNet50        | ❌  | 35.3 M | 60.5 | 91.4 | <a href='https://download.pytorch.org/models/fcn_resnet50_coco-1167a1af.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#fcn_resnet50'>Recipe</a> |
| FCN-ResNet50        | ✅HPO  | 35.3 M | 60.9 | 91.6 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/semantic-segmentation/fcn_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> |
| FCN-ResNet50        | ✅NAS  | 35.3 M | 61.2 | 91.7 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/fcn_resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> |
| FCN-ResNet101       | ❌  | 54.3 M | 63.7 | 91.9 | <a href='https://download.pytorch.org/models/fcn_resnet101_coco-7ecb50ca.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> |
| FCN-ResNet101       | ✅HPO  | 54.3 M | 64.3 | 91.9 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/fcn_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> |
| FCN-ResNet101       | ✅NAS  | 54.3 M | 64.6 | 92.0 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/fcn_resnet101/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> |
| DeepLabV3-ResNet50  | ❌  | 42.0 M | 66.4 | 92.4 | <a href='https://download.pytorch.org/models/deeplabv3_resnet50_coco-cd0a2569.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet50'>Recipe</a> |
| DeepLabV3-ResNet50  | ✅HPO  | 42.0 M | 66.6 | 92.5 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/deeplabv3_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> |
| DeepLabV3-ResNet50  | ✅NAS  | 42.0 M | 66.8 | 92.6 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/deeplabv3_resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> |
| DeepLabV3-ResNet101 | ❌  | 61.0 M | 67.4 | 92.4 | <a href='https://download.pytorch.org/models/deeplabv3_resnet101_coco-586e9e4e.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> |
| DeepLabV3-ResNet101 | ✅HPO  | 61.0 M | 67.8 | 92.5 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/deeplabv3_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> |
| DeepLabV3-ResNet101 | ✅NAS  | 61.0 M | 68.1 | 92.8 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/deeplabv3_resnet101/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> |


`Table 3: Performance comparison of fine-tuned vs. SPG-retrained models across NLP and speech benchmarks.`
- GLUE (Text classification: BERT on CoLA, SST-2, MRPC, QQP, QNLI, and RTE task)
- SQuAD (Question answering: BERT)
- SUPERB (Speech classification: Wav2Vec2 for Audio Classification (AC))

| Task  | SPG  | Metric Type       | Performance (%) | Weights | Command to reproduce |
|-------|------|-------------------|-----------------|---------|----------------------|
| CoLA  | ❌   | Matthews coor     | 56.53           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
| CoLA  | ✅HPO   | Matthews coor     | 62.13           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/cola'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/CoLA-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
| CoLA  | ✅NAS   | Matthews coor     | 63.02           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/cola'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/CoLA-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
| SST-2 | ❌   | Accuracy         | 92.32           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
| SST-2 | ✅HPO   | Accuracy         | 92.54           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/sst2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/SST2-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
| SST-2 | ✅NAS   | Accuracy         | 92.75           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/sst2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/SST2-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
| MRPC  | ❌   | F1/Accuracy      | 88.85/84.09     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
| MRPC  | ✅HPO   | F1/Accuracy      | 91.10/87.25     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/mrpc'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/MRPC-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
| MRPC  | ✅NAS   | F1/Accuracy      | 91.32/87.65     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/mrpc'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/MRPC-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
| QQP   | ❌   | F1/Accuracy      | 87.49/90.71     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
| QQP   | ✅HPO   | F1/Accuracy      | 89.72/90.88     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/qqp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QQP-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
| QQP   | ✅NAS   | F1/Accuracy      | 89.88/91.03     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/qqp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QQP-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
| QNLI  | ❌   | Accuracy         | 90.66           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
| QNLI  | ✅HPO   | Accuracy         | 91.10           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/qnli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QNLI-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
| QNLI  | ✅NAS   | Accuracy         | 91.27           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/qnli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QNLI-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
| RTE   | ❌   | Accuracy         | 65.70           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> |
| RTE   | ✅HPO   | Accuracy         | 72.56           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/rte'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/RTE-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> |
| RTE   | ✅NAS   | Accuracy         | 73.13           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/rte'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/RTE-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> |
| Q/A*  | ❌   | F1/Extra match   | 88.52/81.22     | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-question_answering-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering#fine-tuning-bert-on-squad10'>Recipe</a> |
| Q/A*  | ✅HPO   | F1/Extra match   | 88.67/81.51     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/question-answering/qa'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QA-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/question-answering/run.sh'>run.sh</a> |
| Q/A*  | ✅NAS   | F1/Extra match   | 88.79/81.68     | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/question-answering/qa'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QA-yellow'></a> |  <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/question-answering/run.sh'>run.sh</a> |
| AC†   | ❌   | Accuracy         | 98.26           | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-audio_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification#single-gpu'>Recipe</a> |
| AC†   | ✅HPO   | Accuracy         | 98.31           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/AC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/audio-classification/run.sh'>run.sh</a> |
| AC†   | ✅NAS   | Accuracy         | 98.37           | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/AC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/audio-classification/run.sh'>run.sh</a> |


`Table 4: Performance of SFT vs. SPG-retrained models on GSM8K`


| Model | SPG  | score | Weights | Command to reproduce |
|-------|------|-------|---------|----------------------|
| Gemma-2-2B-it | ❌ | 49.66 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Gemma2B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
| Gemma-2-2B-it | ✅ | 52.31 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Gemma2B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
| Qwen-2.5-0.5B-Instruct | ❌ | 39.12 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Qwen0.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
| Qwen-2.5-0.5B-Instruct | ✅ | 41.70 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Qwen0.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
| Qwen-2.5-1.5B-Instruct | ❌ | 58.68 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Qwen1.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |
| Qwen-2.5-1.5B-Instruct | ✅ | 59.12 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Qwen1.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> |



## Requirements

1. Install `torch>=2.0.0+cu118`.
2. To install other pip packages:
    ```setup
        cd examples
        pip install -r requirements.txt
    ```
3. Prepare the [ImageNet](http://image-net.org/) dataset manually and place it in `/path/to/imagenet`. For image classification examples, pass the argument `--data-path=/path/to/imagenet` to the training script. The extracted dataset directory should follow this structure:
    ```setup
    /path/to/imagenet/:
        train/:
            n01440764: 
                n01440764_18.JPEG ...
            n01443537:
                n01443537_2.JPEG ...
        val/:
            n01440764:
                ILSVRC2012_val_00000293.JPEG ...
            n01443537:
                ILSVRC2012_val_00000236.JPEG ...
    ```
4. Prepare the [MS-COCO 2017](https://cocodataset.org/#home) dataset manually and place it in `/path/to/coco`. For semantic segmentation examples, pass the argument `--data-path=/path/to/coco` to the training script. The extracted dataset directory should follow this structure:
    ```setup
    /path/to/coco/:
        annotations:
            many_json_files.json ...
        train2017:
            000000000009.jpg ...
        val2017:
            000000000139.jpg ...
    ```
5. Prepare the [GSM8K](https://huggingface.co/datasets/openai/gsm8k) dataset manually and place it in `/path/to/gsm8k`. For language modeling examples, pass the argument `--data-path=/path/to/gsm8k` to the training script. The extracted dataset directory should follow this structure:
    ```setup
    /path/to/gsm8k/:
        train.parquet
        test.parquet
    ```
6. For [🗣️ Keyword Spotting subset](https://huggingface.co/datasets/s3prl/superb#ks), [Common Language](https://huggingface.co/datasets/speechbrain/common_language), [SQuAD](https://huggingface.co/datasets/rajpurkar/squad), [Common Voice](https://huggingface.co/datasets/legacy-datasets/common_voice), [GLUE](https://gluebenchmark.com/) and [WMT](https://huggingface.co/datasets/wmt/wmt17) datasets, manual downloading is not required — they will be automatically loaded via the Hugging Face Datasets library when running our `audio-classification`, `question-answering`, `speech-recognition`, `text-classification`, or `translation` examples.