TajaKuzman commited on
Commit
d759e37
·
verified ·
1 Parent(s): 5799e06

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -0
README.md CHANGED
@@ -31,6 +31,20 @@ and 0.720 in macro-F1 on a Croatian test set (440 instances from ParlaMint-HR 4.
31
  An additional evaluation on smaller samples from Czech ParlaMint-CZ, Bulgarian ParlaMint-BG and Ukrainian ParlaMint-UA datasets shows
32
  that the model achieves macro-F1 scores of 0.736, 0.75 and 0.805 on these three test datasets, respectively.
33
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
34
 
35
  ## Use
36
 
 
31
  An additional evaluation on smaller samples from Czech ParlaMint-CZ, Bulgarian ParlaMint-BG and Ukrainian ParlaMint-UA datasets shows
32
  that the model achieves macro-F1 scores of 0.736, 0.75 and 0.805 on these three test datasets, respectively.
33
 
34
+ For end use scenarios, we recommend filtering out predictions based on the model's prediction confidence.
35
+
36
+ When the model was applied to the ParlaMint datasets, we annotated instances that were predicted with confidence below 0.60 as "Mix".
37
+
38
+ With this approach, we annotate as Mix:
39
+ - 8.6% of instances in the English test set
40
+ - 11.4% of instances in the Croatian test set
41
+
42
+ Performance of the model on the remaining instances (all instances not annotated as "Mix"):
43
+
44
+ | | micro-F1 | macro-F1 | accuracy |
45
+ |:---|-----------:|-----------:|-----------:|
46
+ | EN | 0.838 | 0.838 | 0.838 |
47
+ | HR | 0.724 | 0.726 | 0.724 |
48
 
49
  ## Use
50