llama : add llama_vocab, functions -> methods, naming

[ggerganov]

This PR refactors struct llama_model and struct llama_vocab related functionality. Moved the tensor data loading to src/llama-model.cpp. The src/llama.cpp now contains primarily graph build logic. struct llama_vocab is now public in the llama API and the respective calls use it instead of struct llama_model. Improved naming consistency in the public API.

Sub-PRs

• llama : add struct llama_vocab to the API  #11156
• hparams : move vocab params to llama_vocab #11159
• vocab : more pimpl #11165
• vocab : minor tokenization optimizations #11160
• lora : update API names #11167
• llama : update API names to use correct prefix #11174

API changes

Multiple naming changes in the llama_context and llama_model API. Old names have been deprecated.

• Add struct llama_vocab to the API

• llama_n_vocab() now accepts struct llama_vocab instead of struct llama_model

• llama_sampler_init_dry() now accepts struct llama_vocab instead of struct llama_model

• The tokenization API now accepts struct llama_vocab instead of struct llama_model

• The sampler API now accepts struct llama_vocab instead of struct llama_model

• Update API names for improved consistency:

    // before
    LLAMA_API int32_t llama_n_ctx_train(const struct llama_model * model);
    LLAMA_API int32_t llama_n_embd     (const struct llama_model * model);
    LLAMA_API int32_t llama_n_layer    (const struct llama_model * model);
    LLAMA_API int32_t llama_n_head     (const struct llama_model * model);

    LLAMA_API int32_t llama_n_vocab    (const struct llama_vocab * vocab);

    // after
    LLAMA_API int32_t llama_model_n_ctx_train(const struct llama_model * model);
    LLAMA_API int32_t llama_model_n_embd     (const struct llama_model * model);
    LLAMA_API int32_t llama_model_n_layer    (const struct llama_model * model);
    LLAMA_API int32_t llama_model_n_head     (const struct llama_model * model);

    LLAMA_API int32_t llama_vocab_n_tokens   (const struct llama_vocab * vocab);

   ...

Adapter API

Multiple naming changes in this API. Skipped the deprecation phase.

• Rename struct llama_control_vector -> struct llama_adapter_cvec
• Rename struct llama_lora_adapter -> struct llama_adapter_lora
• llama_lora_adapter_[verb](ctx, ...) -> llama_[verb]_adapter_lora(ctx, ...)

Migration instructions

Adapting user code to the changes is fairly straightforward:

• Change functions to use the new names
• Call llama_model_get_vocab(model) where the old API required llama_model and the new API requires llama_vocab

[ggerganov]

I'm stumped by this error in the CI:

https://github.com/ggerganov/llama.cpp/actions/runs/12654124026/job/35261257507?pr=11110#step:8:355

Not sure what is causing it. @slaren Do you have any guess what could be the issue?

[slaren]

Looks like a compiler bug, but I cannot reproduce it locally.

[slaren]

I think it's safe to assume that this is a compiler bug. It only happens with clang on Windows, and only on release builds. Changing the generator from "ninja multi-config" to "ninja" fixes it for the arm build, but not with hip or sycl. The destructors that it complains about seem to be the maps in llama_vocab, it may be some failure trying while trying to inline the calls.

[ggerganov]

Thanks!

[ggerganov]

This should be ready to merge. These are 5 PRs refactoring llama_vocab and llama_model with the main goals being to make the implementation more decoupled and avoid intermediate _impl functions when we can have those as methods of the respective classes. The llama_vocab is now also exposed through the public API because I think long term it would be useful to have this separation between the model and the vocabulary.

This set of PRs, together with #11167 and #11174 all introduce API changes, so it would make sense to merge them together to avoid multiple breaking changes.

After this is merged, I will attempt a similar refactoring for llama_context and llama_kv_cache.

[slaren]

Looks good, I think it is very good to settle on C++ OOP style rather than the current mix of C and C++. Some suggestions for further refactoring:

• Move the code from llama_model_load_from_file to the llama_model constructor
• Move llama_vocab::load to the constructor
• struct with private members is not very common, might be better to use class
• I don't see the point of declaring empty destructors, they can be removed or explicitly set to default. Very few classes need a destructor.
• At some point we should abstract eveything needed to model an architecture to a single class (such that each architecture is a subclass of this class)
• After that, llm_type should probably be removed entirely, and each architecture should have its own enum if needed, with a function to return the type as a string (which by default could be "<arch> <params>")

[LostRuins]

vocab.get_add_bos()

probably a typo @ggerganov ?

[ggerganov]

It's ok - it means "get the add_bos flag". The "get" is necessary to disambiguate from the action of adding a BOS vocab.add_bos().

[LostRuins]

@ggerganov what I mean is that you are checking for eos, but using vocab.get_add_bos(). Should it not be vocab.get_add_eos() instead.

[ggerganov]

Oh yes. Thanks for spotting - fixing.