| # SPG: Sequential Policy Gradient: Lightweight Reinforcement Learning for Model Performance | |
| > 🚀 If you're using Jupyter or Colab, you can follow the demo and run it on a single GPU: | |
| [](https://colab.research.google.com/#fileId=https%3A//huggingface.co/UniversalAlgorithmic/SPG/blob/main/demo_nas.ipynb) | |
| ## Model Zoo: Adaptive Hyperparameter Optimization (HPO) via SPG Algorithm | |
| `Table 1: Performance of pre-trained vs. SPG-retrained models on ImageNet-1K` | |
| | Model | SPG | # Params | Acc@1 (%) | Acc@5 (%) | Weights | Command to reproduce | | |
| |-------|------|----------|-----------|-----------|---------|----------------------| | |
| | MobileNet-V2 | ❌ | 3.5 M | 71.878 | 90.286 | <a href='https://download.pytorch.org/models/mobilenet_v2-b0353104.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#mobilenetv2'>Recipe</a> | | |
| | MobileNet-V2 | ✅HPO | 3.5 M | 72.104 | 90.316 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/mobilenetv2/model_32.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> | | |
| | MobileNet-V2 | ✅NAS | 3.5 M | 72.208 | 90.822 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/mobilenetv2/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/mobilenet_v2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> | | |
| | ResNet-50 | ❌ | 25.6 M | 76.130 | 92.862 | <a href='https://download.pytorch.org/models/resnet50-0676ba61.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#resnet'>Recipe</a> | | |
| | ResNet-50 | ✅HPO | 25.6 M | 77.234 | 93.322 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/resnet50/model_35.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> | | |
| | ResNet-50 | ✅NAS | 25.6 M | 80.970 | 95.481 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> | | |
| | EfficientNet-V2-M | ❌ | 54.1 M | 85.112 | 97.156 | <a href='https://download.pytorch.org/models/efficientnet_v2_m-dc08266a.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#efficientnet-v2'>Recipe</a> | | |
| | EfficientNet-V2-M | ✅HPO | 54.1 M | 85.218 | 97.208 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/efficientnet_v2_m/model_7.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/efficientnet_v2_m-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> | | |
| | EfficientNet-V2-M | ✅NAS | 54.1 M | 85.347 | 97.424 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/efficientnet_v2_m/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/efficientnet_v2_m-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> | | |
| | ViT-B16 | ❌ | 86.6 M | 81.072 | 95.318 | <a href='https://download.pytorch.org/models/vit_b_16-c867db91.pth'><img src='https://img.shields.io/badge/PyTorch-IMAGENET1K_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/classification#vit_b_16'>Recipe</a> | | |
| | ViT-B16 | ✅HPO | 86.6 M | 81.092 | 95.304 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/vit_b_16/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/vit_b_16-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/image-classification/run.sh'>run.sh</a> | | |
| | ViT-B16 | ✅NAS | 86.6 M | 81.114 | 95.320 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/vit_b_16/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/vit_b_16-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/image-classification/run.sh'>run.sh</a> | | |
| `Table 2: Performance of pre-trained vs. SPG-retrained models. All models are evaluated a subset of COCO val2017, on the 21 categories that are present in the Pascal VOC dataset.` | |
| | Model | SPG | # Params | mIoU (%) | pixelwise Acc (%) | Weights | Command to reproduce | | |
| |---------------------|-----|----------|------------|---------------------|---------|----------------------| | |
| | FCN-ResNet50 | ❌ | 35.3 M | 60.5 | 91.4 | <a href='https://download.pytorch.org/models/fcn_resnet50_coco-1167a1af.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#fcn_resnet50'>Recipe</a> | | |
| | FCN-ResNet50 | ✅HPO | 35.3 M | 60.9 | 91.6 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/examples/semantic-segmentation/fcn_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> | | |
| | FCN-ResNet50 | ✅NAS | 35.3 M | 61.2 | 91.7 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/fcn_resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> | | |
| | FCN-ResNet101 | ❌ | 54.3 M | 63.7 | 91.9 | <a href='https://download.pytorch.org/models/fcn_resnet101_coco-7ecb50ca.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> | | |
| | FCN-ResNet101 | ✅HPO | 54.3 M | 64.3 | 91.9 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/fcn_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> | | |
| | FCN-ResNet101 | ✅NAS | 54.3 M | 64.6 | 92.0 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/fcn_resnet101/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/fcn_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> | | |
| | DeepLabV3-ResNet50 | ❌ | 42.0 M | 66.4 | 92.4 | <a href='https://download.pytorch.org/models/deeplabv3_resnet50_coco-cd0a2569.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet50'>Recipe</a> | | |
| | DeepLabV3-ResNet50 | ✅HPO | 42.0 M | 66.6 | 92.5 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/deeplabv3_resnet50/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> | | |
| | DeepLabV3-ResNet50 | ✅NAS | 42.0 M | 66.8 | 92.6 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/deeplabv3_resnet50/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet50-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> | | |
| | DeepLabV3-ResNet101 | ❌ | 61.0 M | 67.4 | 92.4 | <a href='https://download.pytorch.org/models/deeplabv3_resnet101_coco-586e9e4e.pth'><img src='https://img.shields.io/badge/PyTorch-COCO_WITH_VOC_LABELS_V1-FFA500?style=flat&logo=pytorch&logoColor=orange&labelColor=00000000'></a> | <a href='https://github.com/pytorch/vision/tree/main/references/segmentation#deeplabv3_resnet101'>Recipe</a> | | |
| | DeepLabV3-ResNet101 | ✅HPO | 61.0 M | 67.8 | 92.5 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/deeplabv3_resnet101/model_4.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/semantic-segmentation/run.sh'>run.sh</a> | | |
| | DeepLabV3-ResNet101 | ✅NAS | 61.0 M | 68.1 | 92.8 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/deeplabv3_resnet101/model.pth'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/deeplabv3_resnet101-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/semantic-segmentation/run.sh'>run.sh</a> | | |
| `Table 3: Performance comparison of fine-tuned vs. SPG-retrained models across NLP and speech benchmarks.` | |
| - GLUE (Text classification: BERT on CoLA, SST-2, MRPC, QQP, QNLI, and RTE task) | |
| - SQuAD (Question answering: BERT) | |
| - SUPERB (Speech classification: Wav2Vec2 for Audio Classification (AC)) | |
| | Task | SPG | Metric Type | Performance (%) | Weights | Command to reproduce | | |
| |-------|------|-------------------|-----------------|---------|----------------------| | |
| | CoLA | ❌ | Matthews coor | 56.53 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> | | |
| | CoLA | ✅HPO | Matthews coor | 62.13 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/cola'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/CoLA-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> | | |
| | CoLA | ✅NAS | Matthews coor | 63.02 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/cola'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/CoLA-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> | | |
| | SST-2 | ❌ | Accuracy | 92.32 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> | | |
| | SST-2 | ✅HPO | Accuracy | 92.54 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/sst2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/SST2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> | | |
| | SST-2 | ✅NAS | Accuracy | 92.75 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/sst2'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/SST2-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> | | |
| | MRPC | ❌ | F1/Accuracy | 88.85/84.09 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> | | |
| | MRPC | ✅HPO | F1/Accuracy | 91.10/87.25 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/mrpc'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/MRPC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> | | |
| | MRPC | ✅NAS | F1/Accuracy | 91.32/87.65 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/mrpc'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/MRPC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> | | |
| | QQP | ❌ | F1/Accuracy | 87.49/90.71 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> | | |
| | QQP | ✅HPO | F1/Accuracy | 89.72/90.88 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/qqp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QQP-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> | | |
| | QQP | ✅NAS | F1/Accuracy | 89.88/91.03 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/qqp'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QQP-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> | | |
| | QNLI | ❌ | Accuracy | 90.66 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> | | |
| | QNLI | ✅HPO | Accuracy | 91.10 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/qnli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QNLI-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> | | |
| | QNLI | ✅NAS | Accuracy | 91.27 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/qnli'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QNLI-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> | | |
| | RTE | ❌ | Accuracy | 65.70 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-text_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification#glue-tasks'>Recipe</a> | | |
| | RTE | ✅HPO | Accuracy | 72.56 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/text-classification/rte'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/RTE-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/text-classification/run.sh'>run.sh</a> | | |
| | RTE | ✅NAS | Accuracy | 73.13 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/text-classification/rte'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/RTE-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/text-classification/run.sh'>run.sh</a> | | |
| | Q/A* | ❌ | F1/Extra match | 88.52/81.22 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-question_answering-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering#fine-tuning-bert-on-squad10'>Recipe</a> | | |
| | Q/A* | ✅HPO | F1/Extra match | 88.67/81.51 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/question-answering/qa'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QA-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/question-answering/run.sh'>run.sh</a> | | |
| | Q/A* | ✅NAS | F1/Extra match | 88.79/81.68 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/question-answering/qa'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/QA-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/question-answering/run.sh'>run.sh</a> | | |
| | AC† | ❌ | Accuracy | 98.26 | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-audio_classification-yellow'></a> | <a href='https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification#single-gpu'>Recipe</a> | | |
| | AC† | ✅HPO | Accuracy | 98.31 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/hpo-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/AC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/hpo-examples/audio-classification/run.sh'>run.sh</a> | | |
| | AC† | ✅NAS | Accuracy | 98.37 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/AC-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/audio-classification/run.sh'>run.sh</a> | | |
| `Table 4: Performance of SFT vs. SPG-retrained models on GSM8K` | |
| | Model | SPG | score | Weights | Command to reproduce | | |
| |-------|------|-------|---------|----------------------| | |
| | Gemma-2-2B-it | ❌ | 49.66 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Gemma2B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> | | |
| | Gemma-2-2B-it | ✅ | 52.31 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Gemma2B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> | | |
| | Qwen-2.5-0.5B-Instruct | ❌ | 39.12 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Qwen0.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> | | |
| | Qwen-2.5-0.5B-Instruct | ✅ | 41.70 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Qwen0.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> | | |
| | Qwen-2.5-1.5B-Instruct | ❌ | 58.68 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SFT/Qwen1.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> | | |
| | Qwen-2.5-1.5B-Instruct | ✅ | 59.12 | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/tree/main/nas-examples/audio-classification/ac'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Huggingface-SPG/Qwen1.5B-yellow'></a> | <a href='https://huggingface.co/UniversalAlgorithmic/SPG/resolve/main/nas-examples/supervised-fine-tuning/run.sh'>run.sh</a> | | |
| ## Requirements | |
| 1. Install `torch>=2.0.0+cu118`. | |
| 2. To install other pip packages: | |
| ```setup | |
| cd examples | |
| pip install -r requirements.txt | |
| ``` | |
| 3. Prepare the [ImageNet](http://image-net.org/) dataset manually and place it in `/path/to/imagenet`. For image classification examples, pass the argument `--data-path=/path/to/imagenet` to the training script. The extracted dataset directory should follow this structure: | |
| ```setup | |
| /path/to/imagenet/: | |
| train/: | |
| n01440764: | |
| n01440764_18.JPEG ... | |
| n01443537: | |
| n01443537_2.JPEG ... | |
| val/: | |
| n01440764: | |
| ILSVRC2012_val_00000293.JPEG ... | |
| n01443537: | |
| ILSVRC2012_val_00000236.JPEG ... | |
| ``` | |
| 4. Prepare the [MS-COCO 2017](https://cocodataset.org/#home) dataset manually and place it in `/path/to/coco`. For semantic segmentation examples, pass the argument `--data-path=/path/to/coco` to the training script. The extracted dataset directory should follow this structure: | |
| ```setup | |
| /path/to/coco/: | |
| annotations: | |
| many_json_files.json ... | |
| train2017: | |
| 000000000009.jpg ... | |
| val2017: | |
| 000000000139.jpg ... | |
| ``` | |
| 5. Prepare the [GSM8K](https://huggingface.co/datasets/openai/gsm8k) dataset manually and place it in `/path/to/gsm8k`. For language modeling examples, pass the argument `--data-path=/path/to/gsm8k` to the training script. The extracted dataset directory should follow this structure: | |
| ```setup | |
| /path/to/gsm8k/: | |
| train.parquet | |
| test.parquet | |
| ``` | |
| 6. For [🗣️ Keyword Spotting subset](https://huggingface.co/datasets/s3prl/superb#ks), [Common Language](https://huggingface.co/datasets/speechbrain/common_language), [SQuAD](https://huggingface.co/datasets/rajpurkar/squad), [Common Voice](https://huggingface.co/datasets/legacy-datasets/common_voice), [GLUE](https://gluebenchmark.com/) and [WMT](https://huggingface.co/datasets/wmt/wmt17) datasets, manual downloading is not required — they will be automatically loaded via the Hugging Face Datasets library when running our `audio-classification`, `question-answering`, `speech-recognition`, `text-classification`, or `translation` examples. | |