Update tokenizer files
#2
by Xenova HF Staff - opened
No description provided.
Sure thing! This is what I got when I loaded the tokenizer with .from_pretrained followed by .save_pretrained directly afterwards. I got a warning saying that it needed to re-convert the tokenizer.
That is an expected warning based on how HF has that implemented currently.
psinger changed pull request status to closed