README.md

Teach BS

Fluency Scoring

For fluency scoring we use a combination of "hard" structural scoring and "soft" probabilistic scoring from a distilled BERT model.

For probabilitistic scoring we begin by tokenizing the sentence and grouping words. Given the sentence That was unbelievable we tokenize

and produce groupings from the offsets

$${\mathcal{G}}_0=\{0\}{\mathcal{G}}_1=\{1\}{\mathcal{G}}_2=\{2,3,4\}\mathrm{.}\backslash mathcal\; G\_0\; =\; \backslash \{0\backslash \}\; \backslash quad\; \backslash mathcal\; G\_1\; =\; \backslash \{1\backslash \}\; \backslash quad\; \backslash mathcal\; G\_2\; =\; \backslash \{2,\; 3,\; 4\backslash \}.$$

We can then mask entire groups and take the average of the log probabilities of each token, negating so that lower probabilities increase loss:

$${\mathrm{\ell }}_w=-\frac{1}{N}\sum _{i\in {\mathcal{G}}_w}\log \mathbb{P}(t_i)\backslash ell\_w\; =\; -\; \backslash frac\; 1N\; \backslash sum\_\{i\; \backslash in\; \backslash mathcal\; G\_w\}\; \backslash log\; \backslash mathbb\; P(t\_i)$$

where $w$ is the index of the relevant word. This penalizes words that are not preferred by the model in context, but along the way will penalize words that may be correctly used but are simply uncommon. We seek to penalize based on the "contextual awkwardness" of some given word so we will compute a rarity score

$${\mathcal{F}}_w=-\log (f_w+{10}^{-12})\backslash mathcal\; F\_w\; =\; -\backslash log\; (f\_w\; +\; 10^\{-12\})$$

where $f_w$ is computed from the python wordfreq package. The funtion word_frequency will return a value $f_w \in [0, 1]$ where $0$ is extremely rare and $1$ is extremely common. Therefore $\mathcal F_w \in [0, \log 10^{-12} \approx 27.6]$ where a higher score means the word is more rare. We use this to compute an adjusted pseudo log loss $(\text{PLL})$ $\mathcal L$

$${\mathcal{L}}_w={\mathrm{\ell }}_w-\alpha {\mathcal{F}}_w\approx {\mathrm{\ell }}_w+\alpha \log f_w\backslash mathcal\; L\_w\; =\; \backslash ell\_w\; -\; \backslash alpha\; \backslash mathcal\; F\_w\; \backslash approx\; \backslash ell\_w\; +\; \backslash alpha\; \backslash log\; f\_w$$

which applies a downward adjustment to the loss for rare words where $\alpha$ is some weight parameter. Our final step is to produce an adjusted pseudolikelihood estimation

$${\mathcal{J}}_{\text{adj}}=\frac{1}{\mathbb{W}}\sum _{w\in \mathbb{W}}{\mathcal{L}}_w\mathrm{.}\backslash mathcal\; J\_\{\backslash text\{adj\}\}\; =\; \backslash frac\; \{1\}\; \{\backslash mathbb\; W\}\; \backslash sum\_\{w\; \backslash in\; \backslash mathbb\; W\}\; \backslash mathcal\; L\_w.$$

We then generate a fluenct score $\textbf{FS}$ from $0$ to $100$ using the logistic function

$${\mathbf{\text{FS}}}_{\mathbb{W}}=\frac{100}{1+\exp (s\times {\mathcal{J}}_{\text{adj}}-m)}\backslash textbf\{FS\}\_\{\backslash mathbb\; W\}\; =\; \backslash frac\{100\}\{1+\backslash exp(s\; \backslash times\; \backslash mathcal\; J\_\{\backslash text\{adj\}\}\; -\; m)\}$$

where $s$ is some steepness factor and $m$ is the midpoint (where the score should be $50$).