Interactive mode/Session

[BlackLotus]

Hey there sorry if this obviously possible/easy, but is it possible right now to use llama.cpp's interactive mode and if so how?

[abetlen]

Hey, yes that's certainly possible. I don't have an example yet but this can be done through the generate method which returns a generator of tokens which you can also pass a new prompt too to continue generation.

[BlackLotus]

Yeah was confused because generate only took the minimal vars. Thought about hacking an interactive mode into Llamas default call itself something like this
BlackLotus/llama-cpp-python@main...example
(DISCLAIMER: hacked together in a minute, untested, just for show, not usable code)
but since this seemed way too hackish I implemented a minimal PoC that does it's own "session management" like stop words.
Anyway thanks for the reply it really helped.

[SagsMug]

I have tried to port the original main from llama.cpp to python here: https://gist.github.com/SagsMug/c39d5f337b0e0faf32b3ff6bc5bf4956
It seems to work okay

[BlackLotus]

Is a session mode (calling it session here an interactive mode where since user interaction like ctrl+c is required probably isn't wanted in a python lib) wanted in this library?
Implementing a continuous mode through the use of a generator is probably a good enough solution the only "problem" I have with it is that I implemented my own parsing/interacting function, but hacking together an interactive mode should be relatively straight forward.... if it is wanted

[BlackLotus]

I have tried to port the original main from llama.cpp to python here: https://gist.github.com/SagsMug/c39d5f337b0e0faf32b3ff6bc5bf4956 It seems to work okay

Could be an example of using it as chat. :) thx for posting similar to my personal hacked together version, but I didn't bother to use hinting. Maybe I will use your "wrapper".

[abetlen]

I have tried to port the original main from llama.cpp to python here: https://gist.github.com/SagsMug/c39d5f337b0e0faf32b3ff6bc5bf4956 It seems to work okay

This is cool, I'd like to include python ports of the llama.cpp examples to give people an idea of how to use the low-level api. Gladly accept any PRs for this as well.

Yeah was confused because generate only took the minimal vars. Thought about hacking an interactive mode into Llamas default call itself something like this BlackLotus/llama-cpp-python@main...example (DISCLAIMER: hacked together in a minute, untested, just for show, not usable code) but since this seemed way too hackish I implemented a minimal PoC that does it's own "session management" like stop words. Anyway thanks for the reply it really helped.

Ahh gotcha, the reason I haven't implemented continuation / interactive mode in the call function is because there are some bugs with continuing a conversation that ended in a stop word (technically the model has seen the stop word even if you haven't returned the text). There was a recent update to the llama.h API that allows you to restore to a previous model state so I'll work on this a bit.

[abetlen]

@BlackLotus this is now implemented in the examples https://github.com/abetlen/llama-cpp-python/blob/main/examples/low_level_api/low_level_api_chat_cpp.py