remove convert-lora-to-ggml.py

[slaren]

Changes such as permutations to the tensors during model conversion makes converting loras from HF PEFT unreliable, so to avoid confusion I think it is better to remove this entirely until this feature is re-evaluated. It is still possible to use loras created with the finetune example.

[icfly2]

If I read the examples/finetune right it states that only llama models are supported. And with the export-lora I'm not having much luck either with a Phi3 based model that I could previously convert without issues with the removed .py script.

What is the plan going forward? #7225?

[jedt]

for those who are looking for the removed script https://gist.github.com/jedt/87fad3f671589e09d3709e33d29817a4

[miaoshouai]

what is the recommanded ways to merge a finetuned lora into base model now?
most of the online documents are refering to use convert-lora-to-ggml and now it is deprecated.

[slaren]

You can merge the model using merge_and_unload and then convert it to gguf with convert-hf-to-gguf.py.

[jgato]

You can merge the model using merge_and_unload and then convert it to gguf with convert-hf-to-gguf.py.

For the ones that we dont know very much about this... do you have any document on how it should be the procedure now the script has been removed?

[rascasoft]

You can merge the model using merge_and_unload and then convert it to gguf with convert-hf-to-gguf.py.

For the ones that we dont know very much about this... do you have any document on how it should be the procedure now the script has been removed?

I feel like this question needs to be answered: what does "using merge_and_unload" means for someone who have previously just used convert-lora-to-ggml.py and now can't do this anymore?

[shreyasrajesh0308]

I think it means once you have completed training your model, you merge your base model with your adapter weights using the model.merge_and_unload() command in hf. This thread has a discussion on it. Hope this helps!

[rascasoft]

Thanks, @shreyasrajesh0308 I saw the thread and what I understood is that these things work at Python level, so "code", instead using the script was more "CI friendly", if I'm explaining myself.
Have you got any suggestions/examples about an entire workflow in python usable inside a pipeline?