Ollama Models and Plugins aka Tools
Models must have been specifically trained in order to be able to use tools (aka plugins in Witsy). While most closed-source models from large providers are trained for that purpose, this is not necessarily the case with open-source models and therefore the models available through Ollama. If you are interested in reading about this topic, you can refer to:
- https://arxiv.org/abs/2302.04761
- https://cameronrwolfe.substack.com/p/teaching-language-models-to-use-tools
The first thing to check is that you are using a model that has been trained to use tools: in the Ollama Models Library, such models carry a purple tools badge. You can also filter the list at the top by clicking the rounded Tools button. Examples:
- Model that supports tools: https://ollama.com/library/qwq
- Model that does not support tools: https://ollama.com/library/gemma3
But despite some Ollama models being advertised as being able to use tools, models are able to do that with more or less success based on the quality of their training, and their inherent capabilities (linked to their size most notably).
In order to use a tool, a model has to be able to do three things:
- Decide to use a tool
- Output the right instructions to trigger the tool
- Leverage the result of the tool execution
The decision is based on the (specificy of the) context and the description of the tools available to him. For instance it might be easier for a model to decide to use a extract_webpage_content tool when prompted to "summarize https://witsyai.com" rather than "summarize the landing page of the website of witsyai" (this is a purely illustrative example, tools might find it as easy in that specific case but you get the point).
The only influence Witsy has for that decision to happen is the quality of the description of the tools: for instance the description of the extract_webpage_content is "Returns the text content a web page given a URL. Use this tool to get detailed information or summarize the content of a web page.". Some tools descriptions are customizable in Settings | Advanced.
In order to tell Witsy to run a tool, the model has to output specific instructions. The exact format of those instructions vary from one vendor to the other (although most comply to the OpenAI specification for that matter). And once again the quality of the training might yield different results here. Specifically with Ollama models, it has been observed that some models may output those instructions as a normal response instead of a tool_call instruction. In that case, you may see a JSON block in the response of the model. There is no way for Witsy to know if the model intended to output a block JSON code as a response or if it failed to generate the correct instructions. Therefore the response of the model would stop there.
There is generally no issues with that. Tools generally return a very simple JSON object that models have no issue analyzing and extracting the contents from. That content is then injected into their context and used to generate their response.
If you want to understand what is happening in your specific context, there are some things you can do.
The first thing is to ask which tools does it have access to. Below is the response provided by a model which sees the tools:
If you get a generic answer not explicitly describing the Witsy plugins, then the model does not "see" the tools. Some models just don't see the tools they are provided and do not give any indication they can't make use of them.
To understand what is going on, you can:
Witsy offers the possibility to disable plugins. It also can automatically disable plugins: after having submitted a first prompt, some providers respond with a very specific error message indicating that tools are not supported. In that case, Witsy will automatically disable the plugins and submit the prompt again. You should see an alert indicating so. If you have clicked "Do not show this again", then obviously, you would not be notified anymore. You can nevertheless check if plugins are enabled in the Model Settings panel accessible through this icon:
.
For, instance OpenAI chatgpt-4o-latest does not support tools, and OpenAI responds with a 404 error that Witsy is able to take action on by disabling the tools, and automatically re-submitting the request for you.
If plugins are enabled and you want to check what is sent to the model, you can open the Debug Console (link at the bottom of the Model Settings panel). You should be able to find the Request Body sent to the model and should see something like this (five tools in that example, your mileage may vary depending on which plugins you enabled):
When a tool successfully uses tools, you should see two requests made to the model:
- The first one with the original prompt to which the model has responded with
tool_callinstructions - The second one which includes the result of the tool execution:
In summary, while many models, especially those from large providers, are trained to use tools effectively, the ability to utilize them varies based on the model’s training, size, and inherent capabilities. The key steps for a model to successfully use a tool involve deciding to use it, triggering the correct instructions, and leveraging the results to generate a meaningful response. By understanding these aspects and selecting models that are specifically trained for tool usage, users can enhance the effectiveness of their interactions with AI tools. Always consider the quality of training, tool descriptions, and the model's inherent strengths when choosing a model for your needs.