patrickvonplaten/superb-heads

This repo contains all the heads necessary to run the superb downstream models.

Goal: Convert all fine-tuned heads to HF Wav2Vec2 and HuBERT models.

Steps:

1. Make sure model evaluation runs correctly as explained here - it seems like a certain git commit should be used to avoid known bugs - we should probs sync with the authors again on this. The run_downstream.py file corresponds to this script: https://github.com/s3prl/s3prl/blob/master/s3prl/run_downstream.py
2. Debug the evaluation code to see how the model architectures look like for each task.
3. Add the different model architectures to Wav2Vec2 and HuBERT - the architectures are probably called something like Wav2Vec2ForSequenceClassification, etc... -> The idea is to cluster all tasks to the minimum amount of heads (most tasks will probs have the same head architecture, called ...ForSequenceClassification
4. Convert the weights to work correctly with those architectures
5. Test all fine-tuned downstream models to make sure they work correctly.
6. Push converted weights to the hub and merge PR.