Spaces:
Runtime error
Runtime error
File size: 2,010 Bytes
c6d7c4c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
SanityMeaningOfLife:
Signature: "meaning_of_life()"
Input: "with no inputs"
Output: "a single integer, the answer to life the universe and everything"
Fact: "the answer to life the universe and everything is 42"
Description: "Scalar function with no arguments, see if the model can use its internal knowledge to both code and summon a fact from pop culture."
Checks:
input_name:
assert: "f.name"
eq: "meaning_of_life"
no_arguments:
assert: "len(f.args)"
eq: 0
h2g2:
assert: "f.call()"
eq: 42
SanityList:
Signature: "things()"
Input: "with no inputs"
Output: "a list with three values: the number 5, the string 'foobar', the capital city of Spain"
Fact: "the capital city of Spain is Madrid"
Description: "List function, see if the model can combine input facts with internal knowledge."
Checks:
input_name:
assert: "f.name"
eq: "things"
no_arguments:
assert: "len(f.args)"
eq: 0
returns:
assert: "f.call()"
eq: [5, "foobar", "Madrid"]
SanitySecretIdentityMap:
Signature: "secretidentities()"
Input: "with no inputs"
Output: "an object mapping the superheroes Superman, Batman and Spider-Man to their real names"
Fact: "Superman's real name is Clark Kent. Batman's real name is Bruce Wayne. Spider-Man's real name is Peter Parker."
Description: "Object function, see if the model can use its internal knowledge to build a map."
Checks:
input_name:
assert: "f.name"
eq: "secretidentities"
no_arguments:
assert: "len(f.args)"
eq: 0
superman:
assert: "f.call().get('Superman')"
eq: "Clark Kent"
batman:
assert: "f.call().get('Batman')"
eq: "Bruce Wayne"
spiderman:
assert: "f.call().get('Spider-Man')"
eq: "Peter Parker" |