It seems that the add_special flag of llama_tokenize_ will also add EOS, which doesn't seem right here. What would be the way to add only BOS?

main adds BOS manually in case of an empty prompt, so I guess the same can be done here - manually putting BOS and EOS whenever needed.

However, maybe it's better to refactor llama-vocab because llama_tokenize_internal doesn't make a distinction between EOS and BOS.

UPD: the issue with EOS happens in main too, because, for example, in Mistral models EOS should only be put at the end of the answer and never at the end of the prompt. However, initial tokenizing of the prompt may add EOS regardless.

What would be the way to add only BOS?

Should check if llama_add_bos_token(model) is true and add it manually, combined with llama_tokenize(add_special = false).

Is there any case where add_special = true makes sense? Why would anybody want to add a EOS to a prompt?

I was about to say that server might use it because it processes all messages (so, both prompts and responses, and responses might actually need EOS), but I see all llama_tokenize in it with add_special = false.

Is there any case where add_special = true makes sense? Why would anybody want to add a EOS to a prompt?

I think it kind of makes sense for embedding models. add_bos was changed to add_special here: #6498

I was about to say that server might use it because it processes all messages (so, both prompts and responses, and responses might actually need EOS)

No, if the model needs EOS in responses, it will generate it. So this option is not useful for chat use cases.

if the model needs EOS in responses, it will generate it

Was there at any point (back in llama2 days or even earlier) problem with models not generating EOS tokens? There's a guard for antiprompt generated by the model in main.

I think it kind of makes sense for embedding models.

If this only makes sense for embeddings models, wouldn't this make the way most examples use it wrong? I took a quick look, and almost every case still treats this as an add_bos.

I wouldn't be surprised if there are incorrect usages of the llama_tokenize() in the examples. But the general logic that should be followed with the current implementation seems to be:

• If you are tokenizing a full prompt (i.e. starting from the beginning without prior tokens in the context) then use llama_tokenize(add_special = true) and hope that the GGUF KV meta data has correctly exported LLM_KV_TOKENIZER_ADD_BOS and LLM_KV_TOKENIZER_ADD_EOS. For example, both are true for embedding models and the latter is false for chat models.

• If you are tokenizing a portion of a prompt, always use llama_tokenize(add_special = false)

So with this logic in mind, the current implementation in this branch seems OK. Is it failing with a certain model that you tried? If yes, what are the 2 KV values set to?

Is it failing with a certain model that you tried?

No, the responses generated look fine. I missed that it is only added if LLM_KV_TOKENIZER_ADD_EOS is true.