How to use wikilangs/tok with fastText:
from huggingface_hub import hf_hub_download import fasttext model = fasttext.load_model(hf_hub_download("wikilangs/tok", "model.bin"))
https://huggingface.co/wikilangs/tok/resolve/main/visualizations/tokenizer_total_tokens.png
hf download hf://wikilangs/tok/visualizations/tokenizer_total_tokens.png
curl -L -o tokenizer_total_tokens.png https://huggingface.co/wikilangs/tok/resolve/main/visualizations/tokenizer_total_tokens.png
