feat: add reproducible research details

Files changed (25) hide show

README.md +80 -56
example.png +0 -0
paper/dihard3_custom_split/development.txt +62 -0
paper/dihard3_custom_split/train.txt +192 -0
paper/expected_outputs/osd/AMI.development.rttm +0 -0
paper/expected_outputs/osd/AMI.test.rttm +0 -0
paper/expected_outputs/osd/DIHARD.development.rttm +0 -0
paper/expected_outputs/osd/DIHARD.test.rttm +0 -0
paper/expected_outputs/osd/VoxConverse.development.rttm +1617 -0
paper/expected_outputs/osd/VoxConverse.test.rttm +0 -0
paper/expected_outputs/rsg/AMI.development.rttm +0 -0
paper/expected_outputs/rsg/AMI.test.rttm +0 -0
paper/expected_outputs/rsg/DIHARD.development.rttm +0 -0
paper/expected_outputs/rsg/DIHARD.test.rttm +0 -0
paper/expected_outputs/rsg/VoxConverse.development.rttm +0 -0
paper/expected_outputs/vad/AMI.development.rttm +0 -0
paper/expected_outputs/vad/AMI.test.rttm +0 -0
paper/expected_outputs/vad/DIHARD.development.rttm +0 -0
paper/expected_outputs/vad/DIHARD.test.rttm +0 -0
paper/expected_outputs/vad/VoxConverse.development.rttm +0 -0
paper/expected_outputs/vad/VoxConverse.test.rttm +0 -0
paper/expected_outputs/vbx/AMI.rttm +0 -0
paper/expected_outputs/vbx/DIHARD.rttm +0 -0
paper/expected_outputs/vbx/VoxConverse.rttm +0 -0
paper/report.pdf +3 -0

README.md CHANGED Viewed

@@ -19,33 +19,78 @@ inference: false
 # pyannote.audio // speaker segmentation
-This model relies on `pyannote.audio` 2.0 (which is still in development):
 ```bash
 $ pip install https://github.com/pyannote/pyannote-audio/archive/develop.zip
 ```
-## Basic inference
 ```python
->>> from pyannote.audio import Inference
->>> inference = Inference("pyannote/segmentation")
->>> segmentation = inference("audio.wav")
 ```
-## Advanced pipelines
 ### Voice activity detection
 ```python
->>> from pyannote.audio.pipelines import VoiceActivityDetection
->>> HYPER_PARAMETERS = {"onset": 0.5, "offset": 0.5, "min_duration_on": 0.0, "min_duration_off": 0.0}
->>> pipeline = VoiceActivityDetection(segmentation="pyannote/segmentation")
->>> pipeline.instantiate(HYPER_PARAMETERS)
->>> vad = pipeline("audio.wav")
 ```
-In order to reproduce results of the paper, one should use the following hyper-parameter values:
 Dataset         | `onset` | `offset` | `min_duration_on` | `min_duration_off`
 ----------------|---------|----------|-------------------|-------------------
@@ -53,16 +98,18 @@ AMI Mix-Headset | 0.851   | 0.430    | 0.115             | 0.146
 DIHARD3         | 0.855   | 0.292    | 0.036             | 0.001
 VoxConverse     | 0.883   | 0.688    | 0.106             | 0.526
 ### Overlapped speech detection
 ```python
->>> from pyannote.audio.pipelines import OverlappedSpeechDetection
->>> pipeline = OverlappedSpeechDetection(segmentation="pyannote/segmentation")
->>> pipeline.instantiate(HYPER_PARAMETERS)
->>> osd = pipeline("audio.wav")
 ```
-In order to reproduce results of the paper, one should use the following hyper-parameter values:
 Dataset         | `onset` | `offset` | `min_duration_on` | `min_duration_off`
 ----------------|---------|----------|-------------------|-------------------
@@ -70,59 +117,36 @@ AMI Mix-Headset | 0.552   | 0.311    | 0.131             | 0.180
 DIHARD3         | 0.564   | 0.264    | 0.158             | 0.080
 VoxConverse     | 0.617   | 0.387    | 0.367             | 0.334
-### Segmentation
 ```python
->>> from pyannote.audio.pipelines import Segmentation
->>> pipeline = Segmentation(segmentation="pyannote/segmentation")
->>> pipeline.instantiate(HYPER_PARAMETERS)
->>> seg = pipeline("audio.wav")
 ```
-In order to reproduce results of the paper, one should use the following hyper-parameter values:
 Dataset         | `onset` | `offset` | `min_duration_on` | `min_duration_off`
 ----------------|---------|----------|-------------------|-------------------
-AMI Mix-Headset | 0.784   | 0.661    | 0.127             | 0.003
-DIHARD3         | 0.848   | 0.495    | 0.056             | 0.000
-VoxConverse     | 0.882   | 0.779    | 0.304             | 0.484
-### Resegmentation
-```python
->>> from pyannote.audio.pipelines import Resegmentation
->>> pipeline = Resegmentation(segmentation="pyannote/segmentation",
-...                           diarization="baseline")
->>> pipeline.instantiate(HYPER_PARAMETERS)
-```
-VBx RTTM files are also provided in this repository for convenience:
 ```python
->>> from pyannote.database.utils import load_rttm
->>> vbx = load_rttm("/path/to/vbx.rttm")
->>> resegmented_vbx = pipeline({"audio": "DH_EVAL_000.wav",
-...                             "baseline": vbx["DH_EVAL_000"]})
 ```
-In order to reproduce (VBx) results of the paper, one should use the following hyper-parameter values:
-Dataset         | `onset` | `offset` | `min_duration_on` | `min_duration_off`
-----------------|---------|----------|-------------------|-------------------
-AMI Mix-Headset | 0.542   | 0.527    | 0.044             | 0.705
-DIHARD3         | 0.592   | 0.489    | 0.163             | 0.182
-VoxConverse     | 0.537   | 0.724    | 0.410             | 0.563
-## Citations
-```bibtex
-@inproceedings{Bredin2020,
-  Title = {{pyannote.audio: neural building blocks for speaker diarization}},
-  Author = {{Bredin}, Herv{\'e} and {Yin}, Ruiqing and {Coria}, Juan Manuel and {Gelly}, Gregory and {Korshunov}, Pavel and {Lavechin}, Marvin and {Fustes}, Diego and {Titeux}, Hadrien and {Bouaziz}, Wassim and {Gill}, Marie-Philippe},
-  Booktitle = {ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing},
-  Address = {Barcelona, Spain},
-  Month = {May},
-  Year = {2020},
-}
-```

 # pyannote.audio // speaker segmentation
+This model is described in the technical report *[End-to-end speaker segmentation for overlap-aware resegmentation](paper/report.pdf)*, by Hervé Bredin and Antoine Laurent.
+![Example](example.png)
+## Citation
+If you use this model for academic research, please consider citing the `pyannote.audio` library:
+```bibtex
+@inproceedings{Bredin2020,
+  Title = {{pyannote.audio: neural building blocks for speaker diarization}},
+  Author = {{Bredin}, Herv{\'e} and {Yin}, Ruiqing and {Coria}, Juan Manuel and {Gelly}, Gregory and {Korshunov}, Pavel and {Lavechin}, Marvin and {Fustes}, Diego and {Titeux}, Hadrien and {Bouaziz}, Wassim and {Gill}, Marie-Philippe},
+  Booktitle = {ICASSP 2020, IEEE International Conference on Acoustics, Speech, and Signal Processing},
+  Address = {Barcelona, Spain},
+  Month = {May},
+  Year = {2020},
+}
+```
+## Support
+If you (would like to) use this model in commercial products and need help to make the most of it, please contact [me](mailto:[email protected]).
+## Requirements
+This model relies on `pyannote.audio` 2.0 (which is still in development as of April 2nd, 2021):
 ```bash
 $ pip install https://github.com/pyannote/pyannote-audio/archive/develop.zip
 ```
+## Basic usage
 ```python
+from pyannote.audio import Inference
+inference = Inference("pyannote/segmentation")
+segmentation = inference("audio.wav")
+# `segmentation` is a pyannote.core.SlidingWindowFeature
+# instance containing raw segmentation scores like the
+# one pictured above (output)
+from pyannote.audio.pipelines import Segmentation
+pipeline = Segmentation(segmentation="pyannote/segmentation")
+HYPER_PARAMETERS = {
+  # onset/offset activation thresholds
+  "onset": 0.5, "offset": 0.5,
+  # remove speaker turn shorter than that many seconds.
+  "min_duration_on": 0.0,
+  # fill within speaker pauses shorter than that many seconds.
+  "min_duration_off": 0.0
+}
+pipeline.instantiate(HYPER_PARAMETERS)
+segmentation = pipeline("audio.wav")
+# `segmentation` now is a pyannote.core.Annotation
+# instance containing a hard binary segmentation
+# like the one picutred above (reference)
 ```
+## Advanced usage
 ### Voice activity detection
 ```python
+from pyannote.audio.pipelines import VoiceActivityDetection
+pipeline = VoiceActivityDetection(segmentation="pyannote/segmentation")
+pipeline.instantiate(HYPER_PARAMETERS)
+vad = pipeline("audio.wav")
 ```
+In order to reproduce results of the [technical report](paper/report.pdf), one should use the following hyper-parameter values:
 Dataset         | `onset` | `offset` | `min_duration_on` | `min_duration_off`
 ----------------|---------|----------|-------------------|-------------------
 DIHARD3         | 0.855   | 0.292    | 0.036             | 0.001
 VoxConverse     | 0.883   | 0.688    | 0.106             | 0.526
+We also provide the [expected output](paper/expected_outputs/vad) on those three datasets in RTTM format.
 ### Overlapped speech detection
 ```python
+from pyannote.audio.pipelines import OverlappedSpeechDetection
+pipeline = OverlappedSpeechDetection(segmentation="pyannote/segmentation")
+pipeline.instantiate(HYPER_PARAMETERS)
+osd = pipeline("audio.wav")
 ```
+In order to reproduce results of the [technical report](paper/report.pdf), one should use the following hyper-parameter values:
 Dataset         | `onset` | `offset` | `min_duration_on` | `min_duration_off`
 ----------------|---------|----------|-------------------|-------------------
 DIHARD3         | 0.564   | 0.264    | 0.158             | 0.080
 VoxConverse     | 0.617   | 0.387    | 0.367             | 0.334
+We also provide the [expected output](paper/expected_outputs/osd) on those three datasets in RTTM format.
+### Resegmentation
 ```python
+from pyannote.audio.pipelines import Resegmentation
+pipeline = Resegmentation(segmentation="pyannote/segmentation",
+                          diarization="baseline")
+pipeline.instantiate(HYPER_PARAMETERS)
 ```
+In order to reproduce (VBx) results of the technical report, one should use the following hyper-parameter values:
 Dataset         | `onset` | `offset` | `min_duration_on` | `min_duration_off`
 ----------------|---------|----------|-------------------|-------------------
+AMI Mix-Headset | 0.542   | 0.527    | 0.044             | 0.705
+DIHARD3         | 0.592   | 0.489    | 0.163             | 0.182
+VoxConverse     | 0.537   | 0.724    | 0.410             | 0.563
+[VBx RTTM files](paper/expected_outputs/vbx) are also provided in this repository for convenience:
 ```python
+from pyannote.database.utils import load_rttm
+vbx = load_rttm("paper/expected_outputs/vbx/DIHARD.rttm")
+resegmented_vbx = pipeline({"audio": "DH_EVAL_000.wav",
+                            "baseline": vbx["DH_EVAL_000"]})
 ```
+We also provide the [expected output](paper/expected_outputs/rsg) on those three datasets in RTTM format.

example.png ADDED Viewed

paper/dihard3_custom_split/development.txt ADDED Viewed

	@@ -0,0 +1,62 @@

+DH_DEV_0001
+DH_DEV_0023
+DH_DEV_0071
+DH_DEV_0078
+DH_DEV_0002
+DH_DEV_0015
+DH_DEV_0039
+DH_DEV_0003
+DH_DEV_0004
+DH_DEV_0006
+DH_DEV_0010
+DH_DEV_0012
+DH_DEV_0016
+DH_DEV_0024
+DH_DEV_0028
+DH_DEV_0033
+DH_DEV_0041
+DH_DEV_0042
+DH_DEV_0044
+DH_DEV_0047
+DH_DEV_0048
+DH_DEV_0056
+DH_DEV_0005
+DH_DEV_0008
+DH_DEV_0009
+DH_DEV_0017
+DH_DEV_0018
+DH_DEV_0007
+DH_DEV_0030
+DH_DEV_0037
+DH_DEV_0011
+DH_DEV_0027
+DH_DEV_0062
+DH_DEV_0013
+DH_DEV_0022
+DH_DEV_0025
+DH_DEV_0029
+DH_DEV_0031
+DH_DEV_0038
+DH_DEV_0046
+DH_DEV_0058
+DH_DEV_0014
+DH_DEV_0021
+DH_DEV_0026
+DH_DEV_0036
+DH_DEV_0043
+DH_DEV_0049
+DH_DEV_0055
+DH_DEV_0060
+DH_DEV_0068
+DH_DEV_0076
+DH_DEV_0077
+DH_DEV_0090
+DH_DEV_0032
+DH_DEV_0035
+DH_DEV_0065
+DH_DEV_0034
+DH_DEV_0123
+DH_DEV_0133
+DH_DEV_0051
+DH_DEV_0052
+DH_DEV_0054

paper/dihard3_custom_split/train.txt ADDED Viewed

	@@ -0,0 +1,192 @@

+DH_DEV_0080
+DH_DEV_0102
+DH_DEV_0122
+DH_DEV_0144
+DH_DEV_0145
+DH_DEV_0173
+DH_DEV_0176
+DH_DEV_0195
+DH_DEV_0214
+DH_DEV_0221
+DH_DEV_0225
+DH_DEV_0246
+DH_DEV_0066
+DH_DEV_0079
+DH_DEV_0103
+DH_DEV_0113
+DH_DEV_0124
+DH_DEV_0141
+DH_DEV_0206
+DH_DEV_0216
+DH_DEV_0230
+DH_DEV_0059
+DH_DEV_0063
+DH_DEV_0064
+DH_DEV_0070
+DH_DEV_0072
+DH_DEV_0074
+DH_DEV_0075
+DH_DEV_0081
+DH_DEV_0082
+DH_DEV_0084
+DH_DEV_0091
+DH_DEV_0094
+DH_DEV_0098
+DH_DEV_0100
+DH_DEV_0106
+DH_DEV_0109
+DH_DEV_0114
+DH_DEV_0116
+DH_DEV_0126
+DH_DEV_0127
+DH_DEV_0128
+DH_DEV_0130
+DH_DEV_0135
+DH_DEV_0137
+DH_DEV_0138
+DH_DEV_0142
+DH_DEV_0143
+DH_DEV_0156
+DH_DEV_0166
+DH_DEV_0174
+DH_DEV_0178
+DH_DEV_0186
+DH_DEV_0191
+DH_DEV_0196
+DH_DEV_0209
+DH_DEV_0223
+DH_DEV_0234
+DH_DEV_0235
+DH_DEV_0237
+DH_DEV_0238
+DH_DEV_0239
+DH_DEV_0240
+DH_DEV_0241
+DH_DEV_0248
+DH_DEV_0251
+DH_DEV_0252
+DH_DEV_0019
+DH_DEV_0020
+DH_DEV_0045
+DH_DEV_0050
+DH_DEV_0053
+DH_DEV_0073
+DH_DEV_0087
+DH_DEV_0089
+DH_DEV_0096
+DH_DEV_0132
+DH_DEV_0139
+DH_DEV_0167
+DH_DEV_0177
+DH_DEV_0184
+DH_DEV_0201
+DH_DEV_0210
+DH_DEV_0228
+DH_DEV_0236
+DH_DEV_0040
+DH_DEV_0057
+DH_DEV_0105
+DH_DEV_0108
+DH_DEV_0163
+DH_DEV_0188
+DH_DEV_0193
+DH_DEV_0204
+DH_DEV_0208
+DH_DEV_0212
+DH_DEV_0217
+DH_DEV_0093
+DH_DEV_0146
+DH_DEV_0154
+DH_DEV_0155
+DH_DEV_0158
+DH_DEV_0183
+DH_DEV_0187
+DH_DEV_0229
+DH_DEV_0245
+DH_DEV_0061
+DH_DEV_0067
+DH_DEV_0069
+DH_DEV_0083
+DH_DEV_0086
+DH_DEV_0088
+DH_DEV_0118
+DH_DEV_0121
+DH_DEV_0147
+DH_DEV_0153
+DH_DEV_0160
+DH_DEV_0164
+DH_DEV_0165
+DH_DEV_0168
+DH_DEV_0172
+DH_DEV_0181
+DH_DEV_0192
+DH_DEV_0194
+DH_DEV_0202
+DH_DEV_0203
+DH_DEV_0218
+DH_DEV_0250
+DH_DEV_0253
+DH_DEV_0254
+DH_DEV_0095
+DH_DEV_0097
+DH_DEV_0099
+DH_DEV_0104
+DH_DEV_0107
+DH_DEV_0110
+DH_DEV_0111
+DH_DEV_0115
+DH_DEV_0117
+DH_DEV_0129
+DH_DEV_0148
+DH_DEV_0149
+DH_DEV_0150
+DH_DEV_0151
+DH_DEV_0152
+DH_DEV_0157
+DH_DEV_0161
+DH_DEV_0171
+DH_DEV_0182
+DH_DEV_0185
+DH_DEV_0190
+DH_DEV_0197
+DH_DEV_0198
+DH_DEV_0200
+DH_DEV_0205
+DH_DEV_0213
+DH_DEV_0215
+DH_DEV_0219
+DH_DEV_0220
+DH_DEV_0222
+DH_DEV_0224
+DH_DEV_0231
+DH_DEV_0242
+DH_DEV_0243
+DH_DEV_0247
+DH_DEV_0249
+DH_DEV_0092
+DH_DEV_0112
+DH_DEV_0119
+DH_DEV_0131
+DH_DEV_0136
+DH_DEV_0175
+DH_DEV_0199
+DH_DEV_0207
+DH_DEV_0232
+DH_DEV_0134
+DH_DEV_0159
+DH_DEV_0169
+DH_DEV_0170
+DH_DEV_0180
+DH_DEV_0189
+DH_DEV_0211
+DH_DEV_0226
+DH_DEV_0244
+DH_DEV_0085
+DH_DEV_0101
+DH_DEV_0120
+DH_DEV_0125
+DH_DEV_0140
+DH_DEV_0162
+DH_DEV_0179
+DH_DEV_0227
+DH_DEV_0233

paper/expected_outputs/osd/AMI.development.rttm ADDED Viewed