Update README.md
Browse files
README.md
CHANGED
@@ -31,6 +31,20 @@ and 0.720 in macro-F1 on a Croatian test set (440 instances from ParlaMint-HR 4.
|
|
31 |
An additional evaluation on smaller samples from Czech ParlaMint-CZ, Bulgarian ParlaMint-BG and Ukrainian ParlaMint-UA datasets shows
|
32 |
that the model achieves macro-F1 scores of 0.736, 0.75 and 0.805 on these three test datasets, respectively.
|
33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
34 |
|
35 |
## Use
|
36 |
|
|
|
31 |
An additional evaluation on smaller samples from Czech ParlaMint-CZ, Bulgarian ParlaMint-BG and Ukrainian ParlaMint-UA datasets shows
|
32 |
that the model achieves macro-F1 scores of 0.736, 0.75 and 0.805 on these three test datasets, respectively.
|
33 |
|
34 |
+
For end use scenarios, we recommend filtering out predictions based on the model's prediction confidence.
|
35 |
+
|
36 |
+
When the model was applied to the ParlaMint datasets, we annotated instances that were predicted with confidence below 0.60 as "Mix".
|
37 |
+
|
38 |
+
With this approach, we annotate as Mix:
|
39 |
+
- 8.6% of instances in the English test set
|
40 |
+
- 11.4% of instances in the Croatian test set
|
41 |
+
|
42 |
+
Performance of the model on the remaining instances (all instances not annotated as "Mix"):
|
43 |
+
|
44 |
+
| | micro-F1 | macro-F1 | accuracy |
|
45 |
+
|:---|-----------:|-----------:|-----------:|
|
46 |
+
| EN | 0.838 | 0.838 | 0.838 |
|
47 |
+
| HR | 0.724 | 0.726 | 0.724 |
|
48 |
|
49 |
## Use
|
50 |
|