Tokenizer class IndoNLGTokenizer does not exist or is not currently imported.
#3
by Maki21 - opened
I want to try and use this model for my research but i can't load the tokenizer its just appear error that say :
Tokenizer class IndoNLGTokenizer does not exist or is not currently imported.
i ask someone in github and they say :
You should try to ask the other of the model on the community tab how to use it
Hi @Maki21 ,
To use the tokenizer you can use the indobenchmark-toolkit pip package. We couldn't load it with the standard tokenizer since, back then, we make some modification to the tokenization code. You can check how we use the tokenizer on the examples folder of the indonlg repo.
Basically, you can initialize the tokenizer in this way:
from indobenchmark import IndoNLGTokenizer
tokenizer = IndoNLGTokenizer.from_pretrained('indobenchmark/indobart-v2')
Hope it helps!
samuelcahyawijaya changed discussion status to closed