EOS BOS both <|endoftext|>?
#23
by
mattjcly
- opened
https://huggingface.co/microsoft/Phi-4-mini-instruct/blob/main/tokenizer_config.json#L103-L106
I would imagine this is a bug?
I think no. The chat template doesn't include the "BOS" token.