Update Space (evaluate main: 940d6dee)

README.md

@@ -11,10 +11,10 @@ tags:
 - evaluate
 - measurement
 description: >-
-  Perplexity (PPL) can be used for evaluating to what extent a dataset is similar to the distribution of text that a given model was trained on.
-  It is defined as the exponentiated average negative log-likelihood of a sequence.
+  Perplexity (PPL) can be used to evaluate the extent to which a dataset is similar to the distribution of text that a given model was trained on.
+  It is defined as the exponentiated average negative log-likelihood of a sequence, calculated with exponent base `e`.
 
-  For more information, see https://huggingface.co/docs/transformers/perplexity
+  For more information on perplexity, see [this tutorial](https://huggingface.co/docs/transformers/perplexity).
 ---
 
 # Measurement Card for Perplexity
@@ -22,8 +22,10 @@ description: >-
 ## Measurement Description
 Given a model and an input text sequence, perplexity measures how likely the model is to generate the input text sequence.
 
-As a measurement, it can be used to to evaluate how well a selection of texts matches the distribution of text that the input model was trained on.
-In this case, the model input should be a trained model, and the input texts should be the text to be evaluated.
+As a measurement, it can be used to evaluate how well text matches the distribution of text that the input model was trained on.
+In this case, `model_id` should be the trained model, and `data` should be the text to be evaluated.
+
+This implementation of perplexity is calculated with log base `e`, as in `perplexity = e**(sum(losses) / num_tokenized_tokens)`, following recent convention in deep learning frameworks.
 
 ## Intended Uses
 Dataset analysis or exploration.
@@ -35,16 +37,16 @@ The measurement takes a list of texts as input, as well as the name of the model
 ```python
 from evaluate import load
 perplexity = load("perplexity",  module_type= "measurement")
-results = perplexity.compute(input_texts=input_texts, model_id='gpt2')
+results = perplexity.compute(data=input_texts, model_id='gpt2')
 ```
 
 ### Inputs
 - **model_id** (str): model used for calculating Perplexity. NOTE: Perplexity can only be calculated for causal language models.
     - This includes models such as gpt2, causal variations of bert, causal versions of t5, and more (the full list can be found in the AutoModelForCausalLM documentation here: https://huggingface.co/docs/transformers/master/en/model_doc/auto#transformers.AutoModelForCausalLM )
-- **input_texts** (list of str): input text, each separate text snippet is one list entry.
+- **data** (list of str): input text, where each separate text snippet is one list entry.
 - **batch_size** (int): the batch size to run texts through the model. Defaults to 16.
 - **add_start_token** (bool): whether to add the start token to the texts, so the perplexity can include the probability of the first word. Defaults to True.
-- **device** (str): device to run on, defaults to 'cuda' when available
+- **device** (str): device to run on, defaults to `cuda` when available
 
 ### Output Values
 This metric outputs a dictionary with the perplexity scores for the text input in the list, and the average perplexity.
@@ -54,7 +56,7 @@ If one of the input texts is longer than the max input length of the model, then
 {'perplexities': [8.182524681091309, 33.42122268676758, 27.012239456176758], 'mean_perplexity': 22.871995608011883}
 ```
 
-This metric's range is 0 and up. A lower score is better.
+The range of this metric is [0, inf). A lower score is better.
 
 #### Values from Popular Papers
 
@@ -62,17 +64,17 @@ This metric's range is 0 and up. A lower score is better.
 ### Examples
 Calculating perplexity on input_texts defined here:
 ```python
-perplexity = evaluate.load("perplexity", module_type= "measurement")
+perplexity = evaluate.load("perplexity", module_type="measurement")
 input_texts = ["lorem ipsum", "Happy Birthday!", "Bienvenue"]
 results = perplexity.compute(model_id='gpt2',
                              add_start_token=False,
-                             input_texts=input_texts)
+                             data=input_texts)
 print(list(results.keys()))
 >>>['perplexities', 'mean_perplexity']
 print(round(results["mean_perplexity"], 2))
->>>78.22
+>>>646.74
 print(round(results["perplexities"][0], 2))
->>>11.11
+>>>32.25
 ```
 Calculating perplexity on input_texts loaded in from a dataset:
 ```python
@@ -82,13 +84,13 @@ input_texts = datasets.load_dataset("wikitext",
                                     split="test")["text"][:50]
 input_texts = [s for s in input_texts if s!='']
 results = perplexity.compute(model_id='gpt2',
-                             input_texts=input_texts)
+                             data=input_texts)
 print(list(results.keys()))
 >>>['perplexities', 'mean_perplexity']
 print(round(results["mean_perplexity"], 2))
->>>60.35
+>>>576.76
 print(round(results["perplexities"][0], 2))
->>>81.12
+>>>889.28
 ```
 
 ## Limitations and Bias

perplexity.py

@@ -29,7 +29,7 @@ _CITATION = """\
 
 _DESCRIPTION = """
 Perplexity (PPL) can be used for evaluating to what extent a dataset is similar to the distribution of text that a given model was trained on.
-It is defined as the exponentiated average negative log-likelihood of a sequence.
+It is defined as the exponentiated average negative log-likelihood of a sequence, calculated with exponent base `e`.
 
 For more information, see https://huggingface.co/docs/transformers/perplexity
 """
@@ -64,9 +64,9 @@ Examples:
         >>> print(list(results.keys()))
         ['perplexities', 'mean_perplexity']
         >>> print(round(results["mean_perplexity"], 2))
-        78.22
+        646.74
         >>> print(round(results["perplexities"][0], 2))
-        11.11
+        32.25
 
     Example 2:
         >>> from datasets import load_dataset
@@ -78,9 +78,9 @@ Examples:
         >>> print(list(results.keys()))
         ['perplexities', 'mean_perplexity']
         >>> print(round(results["mean_perplexity"], 2)) # doctest: +SKIP
-        60.35
+        576.76
         >>> print(round(results["perplexities"][0], 2)) # doctest: +SKIP
-        81.12
+        889.28
 """
 
 
@@ -180,7 +180,7 @@ class Perplexity(evaluate.Measurement):
             shift_labels = labels[..., 1:].contiguous()
             shift_attention_mask_batch = attn_mask[..., 1:].contiguous()
 
-            perplexity_batch = torch.exp2(
+            perplexity_batch = torch.exp(
                 (loss_fct(shift_logits.transpose(1, 2), shift_labels) * shift_attention_mask_batch).sum(1)
                 / shift_attention_mask_batch.sum(1)
             )

requirements.txt

@@ -1,3 +1,3 @@
-git+https://github.com/huggingface/evaluate@288e417936483b1b48ac2dc64a9d9c80ae0ed7e6
+git+https://github.com/huggingface/evaluate@940d6dee3b4a23eabb0c81e4117c9533cd7c458a
 torch
 transformers