Spaces:
Runtime error
Appropriate tokenizer?
Hello! I was wondering what we mean in with “appropriate tokenizer” in the body of the question for the art tooling residency application.
Does that mean we should use a specific tokenizer, for example one that matches the model checkpoint (distillbert-cased-tokenizer), or is the choice of tokenizer open to more simple choices like using vanilla Python to split the example string at whitespace?
In the first case, I would like to know if we could use the tokenizer with default parameters, or if it is expected that we tweak the input parameters to give specific treatment (e.g to ignore) to funky punctuation characters such as “\’”.
Thanks!
I think the second chapter of this course contains a hint 🤗 :
https://huggingface.co/course
I am also an applicant here and I think with this question, they want to check the understanding of the applicant about their product. Whether an applicant is equipped with the understanding of Huggingface or not and whether they can be able to apply Huggingface utilities on the real world problems.
Thank you.
Best regards.