From 364d684b9ae3c6fe0232e425bc9b429afd4c724c Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Sun, 27 Aug 2023 14:44:20 -0300 Subject: [PATCH 01/18] Improve README.md for building in Termux on Android devices Added direction to build with `make`, and simplified `OpenBlas` method. These instructions should work for any Android device capable of running `llama.cpp`. Clarified on where to _move_ the models in Termux. `llama.cpp` is diverse, so I added a Usage example for new users. I left `CLBlast` instructions unchanged for now. --- README.md | 41 ++++++++++++++++++++++++++++++++--------- 1 file changed, 32 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index bf3eb0b76e1da..50a794a07d71f 100644 --- a/README.md +++ b/README.md @@ -791,17 +791,42 @@ Finally, copy the `llama` binary and the model files to your device storage. Her https://user-images.githubusercontent.com/271616/225014776-1d567049-ad71-4ef2-b050-55b0b3b9274c.mp4 -#### Building the Project using Termux (F-Droid) -Termux from F-Droid offers an alternative route to execute the project on an Android device. This method empowers you to construct the project right from within the terminal, negating the requirement for a rooted device or SD Card. +#### Building the Project in Termux (F-Droid) +[Termux](https://termux.dev/) is an alternative method to run `llama.cpp` on Android devices. -Outlined below are the directives for installing the project using OpenBLAS and CLBlast. This combination is specifically designed to deliver peak performance on recent devices that feature a GPU. +Ensure Termux is up to date and clone the repo: +``` +apt update && apt upgrade +$HOME +git clone https://github.com/ggerganov/llama.cpp +``` + +Build `llama.cpp`: +``` +$HOME +cd llama.cpp +make +``` -If you opt to utilize OpenBLAS, you'll need to install the corresponding package. +It's possible to include OpenBlas while building: ``` -apt install libopenblas +$HOME +pkg install libopenblas +cd llama.cpp +cmake -B build -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS +cd build +cmake --build . --config Release ``` -Subsequently, if you decide to incorporate CLBlast, you'll first need to install the requisite OpenCL packages: +Move your model to the $HOME directory in Termux, for example: +``` +cd storage/downloads +mv 7b-model.gguf ~/ +``` + +Usage example:`./llama.cpp/main -m ~/7b-model.gguf --color -c 2048 --keep -1 -n -2 -b 10 -i -ins` + +Alternatively, to enable CLBlast then install the requisite OpenCL packages: ``` apt install ocl-icd opencl-headers opencl-clhpp clinfo ``` @@ -830,9 +855,7 @@ export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH (Note: some Android devices, like the Zenfone 8, need the following command instead - "export LD_LIBRARY_PATH=/system/vendor/lib64:$LD_LIBRARY_PATH". Source: https://www.reddit.com/r/termux/comments/kc3ynp/opencl_working_in_termux_more_in_comments/ ) -For easy and swift re-execution, consider documenting this final part in a .sh script file. This will enable you to rerun the process with minimal hassle. - -Place your desired model into the `~/llama.cpp/models/` directory and execute the `./main (...)` script. +For easy and swift re-execution, consider documenting this final part in a .sh script file. This will allow you to run `./main (...)` with minimal hassle. ### Docker From cd7b3edeed6a1ee85a6a314652856294fbb53b27 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Sun, 27 Aug 2023 14:51:57 -0300 Subject: [PATCH 02/18] clarified wording --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 50a794a07d71f..160138768ea52 100644 --- a/README.md +++ b/README.md @@ -792,7 +792,7 @@ Finally, copy the `llama` binary and the model files to your device storage. Her https://user-images.githubusercontent.com/271616/225014776-1d567049-ad71-4ef2-b050-55b0b3b9274c.mp4 #### Building the Project in Termux (F-Droid) -[Termux](https://termux.dev/) is an alternative method to run `llama.cpp` on Android devices. +[Termux](https://termux.dev/) is a way to run `llama.cpp` on Android devices. Ensure Termux is up to date and clone the repo: ``` From 83df548b86d7a2e97dbab2552f6ef577fe827513 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Sun, 27 Aug 2023 16:09:13 -0300 Subject: [PATCH 03/18] clean up Usage example made Usage example more clear in the case Users build with cmake --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 160138768ea52..95574df0997c2 100644 --- a/README.md +++ b/README.md @@ -824,7 +824,7 @@ cd storage/downloads mv 7b-model.gguf ~/ ``` -Usage example:`./llama.cpp/main -m ~/7b-model.gguf --color -c 2048 --keep -1 -n -2 -b 10 -i -ins` +Usage example:`./main -m ~/7b-model.gguf --color -c 2048 --keep -1 -n -2 -b 10 -i -ins` Alternatively, to enable CLBlast then install the requisite OpenCL packages: ``` From ee55b1e528aae65eb72b3870211db1bc15a1f904 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Sun, 27 Aug 2023 16:28:28 -0300 Subject: [PATCH 04/18] dummy-proof directions $HOME before make is unneccesary. If User is new with Termux then typing cd storage/downloads before $HOME is confusing. clarified directions. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 95574df0997c2..6b4b9bb999b93 100644 --- a/README.md +++ b/README.md @@ -803,7 +803,6 @@ git clone https://github.com/ggerganov/llama.cpp Build `llama.cpp`: ``` -$HOME cd llama.cpp make ``` @@ -820,6 +819,7 @@ cmake --build . --config Release Move your model to the $HOME directory in Termux, for example: ``` +$HOME cd storage/downloads mv 7b-model.gguf ~/ ``` From e69b7b5a38d3041b20ffe46068cb2a765b24ef6a Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Sun, 27 Aug 2023 16:55:56 -0300 Subject: [PATCH 05/18] corrected navigation $HOME is a script within fish shell, assuming User uses bash then we use `cd` to navigate to /data/data/com.termux/files/home --- README.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 6b4b9bb999b93..b9c578555931a 100644 --- a/README.md +++ b/README.md @@ -797,7 +797,7 @@ https://user-images.githubusercontent.com/271616/225014776-1d567049-ad71-4ef2-b0 Ensure Termux is up to date and clone the repo: ``` apt update && apt upgrade -$HOME +cd git clone https://github.com/ggerganov/llama.cpp ``` @@ -809,7 +809,6 @@ make It's possible to include OpenBlas while building: ``` -$HOME pkg install libopenblas cd llama.cpp cmake -B build -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS @@ -819,7 +818,7 @@ cmake --build . --config Release Move your model to the $HOME directory in Termux, for example: ``` -$HOME +cd cd storage/downloads mv 7b-model.gguf ~/ ``` From 8cb4ee0f4c324f716d78f0af115a9412a5a2ff8c Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Sun, 27 Aug 2023 22:16:45 -0300 Subject: [PATCH 06/18] Clarify move model directions remove reference to Termux `$HOME` path. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b9c578555931a..e505a165770f9 100644 --- a/README.md +++ b/README.md @@ -816,7 +816,7 @@ cd build cmake --build . --config Release ``` -Move your model to the $HOME directory in Termux, for example: +Move your model inside `/data/data/com.termux/files/home`, for example: ``` cd cd storage/downloads From 0a7ef3a5f9e164b3a70239bdd8b4d6f9061f2962 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Mon, 28 Aug 2023 00:17:47 -0300 Subject: [PATCH 07/18] crystal-clear move path destination User needs to move the model to the correct location, and doesn't need to know/care about the absolute path in Termux --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e505a165770f9..4b83daf49de30 100644 --- a/README.md +++ b/README.md @@ -816,7 +816,7 @@ cd build cmake --build . --config Release ``` -Move your model inside `/data/data/com.termux/files/home`, for example: +Move your model to the home directory (~/), for example: ``` cd cd storage/downloads From 8e00404bc3fd320cc1175859eb59da43823a2d56 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Mon, 28 Aug 2023 01:33:56 -0300 Subject: [PATCH 08/18] typo fixed --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 4b83daf49de30..3d6e93dcbcf6f 100644 --- a/README.md +++ b/README.md @@ -816,7 +816,7 @@ cd build cmake --build . --config Release ``` -Move your model to the home directory (~/), for example: +Move your model to the home directory (`~/`), for example: ``` cd cd storage/downloads From 2612b3606179181e9e419988ad1b0452fc6c88b3 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Wed, 30 Aug 2023 21:53:26 -0300 Subject: [PATCH 09/18] Update Usage Example original example was generic, adjusted --prompt to fit alpaca-style --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3d6e93dcbcf6f..9e3ebc6064f2c 100644 --- a/README.md +++ b/README.md @@ -820,10 +820,10 @@ Move your model to the home directory (`~/`), for example: ``` cd cd storage/downloads -mv 7b-model.gguf ~/ +mv 7b-model.gguf.q4_0.bin ~/ ``` -Usage example:`./main -m ~/7b-model.gguf --color -c 2048 --keep -1 -n -2 -b 10 -i -ins` +Usage example:`./main -m ~/7b-model.gguf.q4_0.bin --color -c 2048 --keep -1 -n -2 -b 7 -ins -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.'\n\n'### Instruction:'\n'Hi!'\n\n'### Response:Hi! How may I assist you?' Alternatively, to enable CLBlast then install the requisite OpenCL packages: ``` From 82700d8c38362f81d9c973ca968d58d1d846f736 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Wed, 30 Aug 2023 21:55:25 -0300 Subject: [PATCH 10/18] Update README.md typo fixed --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9e3ebc6064f2c..ffd4ce3f8ad99 100644 --- a/README.md +++ b/README.md @@ -823,7 +823,7 @@ cd storage/downloads mv 7b-model.gguf.q4_0.bin ~/ ``` -Usage example:`./main -m ~/7b-model.gguf.q4_0.bin --color -c 2048 --keep -1 -n -2 -b 7 -ins -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.'\n\n'### Instruction:'\n'Hi!'\n\n'### Response:Hi! How may I assist you?' +Usage example:`./main -m ~/7b-model.gguf.q4_0.bin --color -c 2048 --keep -1 -n -2 -b 7 -ins -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.'\n\n'### Instruction:'\n'Hi!'\n\n'### Response:Hi! How may I assist you?'` Alternatively, to enable CLBlast then install the requisite OpenCL packages: ``` From 32bbb554a75faba38affb494cf06c94471a1b402 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Wed, 30 Aug 2023 22:12:12 -0300 Subject: [PATCH 11/18] Made Usage example more easy to copy also looks better, and consistent --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ffd4ce3f8ad99..233995a31a75c 100644 --- a/README.md +++ b/README.md @@ -823,7 +823,10 @@ cd storage/downloads mv 7b-model.gguf.q4_0.bin ~/ ``` -Usage example:`./main -m ~/7b-model.gguf.q4_0.bin --color -c 2048 --keep -1 -n -2 -b 7 -ins -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.'\n\n'### Instruction:'\n'Hi!'\n\n'### Response:Hi! How may I assist you?'` +Usage example: +``` +./main -m ~/7b-model.gguf.q4_0.bin --color -c 2048 --keep -1 -n -2 -b 7 -ins -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.'\n\n'### Instruction:'\n'Hi!'\n\n'### Response:Hi! How may I assist you?' +``` Alternatively, to enable CLBlast then install the requisite OpenCL packages: ``` From 245a02efce49351bbb65a319cacb95ea36e9ef35 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Thu, 31 Aug 2023 20:52:28 -0300 Subject: [PATCH 12/18] adjust OpenBlas from cmake to make --- README.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/README.md b/README.md index 233995a31a75c..bcd61aa3dd745 100644 --- a/README.md +++ b/README.md @@ -811,9 +811,7 @@ It's possible to include OpenBlas while building: ``` pkg install libopenblas cd llama.cpp -cmake -B build -DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS -cd build -cmake --build . --config Release +make LLAMA_OPENBLAS=1 ``` Move your model to the home directory (`~/`), for example: From 5a3dc43bfe8e50aa118baae3888ed60b9ce86818 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Thu, 31 Aug 2023 22:13:31 -0300 Subject: [PATCH 13/18] Clean/Update Android ClBlast section Replaced outdated, ineffecient steps for a streamlined method. --- README.md | 32 ++++++-------------------------- 1 file changed, 6 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index bcd61aa3dd745..27553c2d5314a 100644 --- a/README.md +++ b/README.md @@ -826,36 +826,16 @@ Usage example: ./main -m ~/7b-model.gguf.q4_0.bin --color -c 2048 --keep -1 -n -2 -b 7 -ins -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.'\n\n'### Instruction:'\n'Hi!'\n\n'### Response:Hi! How may I assist you?' ``` -Alternatively, to enable CLBlast then install the requisite OpenCL packages: +To build with `OpenCL` then install the requisite packages: ``` -apt install ocl-icd opencl-headers opencl-clhpp clinfo -``` - -In order to compile CLBlast, you'll need to first clone the respective Git repository, which can be found at this URL: https://github.com/CNugteren/CLBlast. Alongside this, clone this repository into your home directory. Once this is done, navigate to the CLBlast folder and execute the commands detailed below: -``` -cmake . -make -cp libclblast.so* $PREFIX/lib -cp ./include/clblast.h ../llama.cpp -``` - -Following the previous steps, navigate to the LlamaCpp directory. To compile it with OpenBLAS and CLBlast, execute the command provided below: -``` -cp /data/data/com.termux/files/usr/include/openblas/cblas.h . -cp /data/data/com.termux/files/usr/include/openblas/openblas_config.h . -make LLAMA_CLBLAST=1 //(sometimes you need to run this command twice) -``` - -Upon completion of the aforementioned steps, you will have successfully compiled the project. To run it using CLBlast, a slight adjustment is required: a command must be issued to direct the operations towards your device's physical GPU, rather than the virtual one. The necessary command is detailed below: -``` -GGML_OPENCL_PLATFORM=0 -GGML_OPENCL_DEVICE=0 -export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH +pkg install ocl-icd opencl-headers clblast +cd llama.cpp +make LLAMA_CLBLAST=1 ``` -(Note: some Android devices, like the Zenfone 8, need the following command instead - "export LD_LIBRARY_PATH=/system/vendor/lib64:$LD_LIBRARY_PATH". Source: https://www.reddit.com/r/termux/comments/kc3ynp/opencl_working_in_termux_more_in_comments/ ) +Finally, use `export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH`, to enable GPU then `./main ... -ngl 1` -For easy and swift re-execution, consider documenting this final part in a .sh script file. This will allow you to run `./main (...)` with minimal hassle. +(Note: Use `unset LD_LIBRARY_PATH` to re-link executables) ### Docker From 673f424f34b600b0ed9aeaff129043b8d0eb0c9f Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Thu, 31 Aug 2023 22:18:26 -0300 Subject: [PATCH 14/18] typo fix --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 27553c2d5314a..461b25347c8af 100644 --- a/README.md +++ b/README.md @@ -833,7 +833,7 @@ cd llama.cpp make LLAMA_CLBLAST=1 ``` -Finally, use `export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH`, to enable GPU then `./main ... -ngl 1` +Finally, use `export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH` to enable GPU, then `./main ... -ngl 1` (Note: Use `unset LD_LIBRARY_PATH` to re-link executables) From d719f6cd123ce0d83a49dec689266801a10a6ca2 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Fri, 1 Sep 2023 00:23:28 -0300 Subject: [PATCH 15/18] easily copy/paste export made export better for copy/pasting, looks better --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 461b25347c8af..89f1831afa32a 100644 --- a/README.md +++ b/README.md @@ -826,14 +826,18 @@ Usage example: ./main -m ~/7b-model.gguf.q4_0.bin --color -c 2048 --keep -1 -n -2 -b 7 -ins -p 'Below is an instruction that describes a task. Write a response that appropriately completes the request.'\n\n'### Instruction:'\n'Hi!'\n\n'### Response:Hi! How may I assist you?' ``` -To build with `OpenCL` then install the requisite packages: +For building with `OpenCL` then install the requisite packages: ``` pkg install ocl-icd opencl-headers clblast cd llama.cpp make LLAMA_CLBLAST=1 ``` -Finally, use `export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH` to enable GPU, then `./main ... -ngl 1` +Use the following to enable GPU: +``` +export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH +``` +then `./main ... --gpu-layers 1` (Note: Use `unset LD_LIBRARY_PATH` to re-link executables) From 736ff7c103d6688849a1bc48acb5c8c1b5e5ffd7 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Fri, 1 Sep 2023 00:32:40 -0300 Subject: [PATCH 16/18] stylized OpenBlas text now is correctly matched with `CLBlast` & `llama.cpp` --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 89f1831afa32a..8f80ba524c4b7 100644 --- a/README.md +++ b/README.md @@ -807,7 +807,7 @@ cd llama.cpp make ``` -It's possible to include OpenBlas while building: +It's possible to enable `OpenBlas` while building: ``` pkg install libopenblas cd llama.cpp From b47ee695727b2f93ad7767543d71d54c3605a5dc Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Fri, 1 Sep 2023 02:19:47 -0300 Subject: [PATCH 17/18] Ensure device compatibility for CLBlast Some devices path from Vendor, others path from System --- README.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8f80ba524c4b7..14687c2719d68 100644 --- a/README.md +++ b/README.md @@ -833,13 +833,17 @@ cd llama.cpp make LLAMA_CLBLAST=1 ``` -Use the following to enable GPU: +Use one of the following to enable GPU: ``` export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH ``` +or +``` +export LD_LIBRARY_PATH=/system/vendor/lib64:$LD_LIBRARY_PATH +``` then `./main ... --gpu-layers 1` -(Note: Use `unset LD_LIBRARY_PATH` to re-link executables) +(Note: Use `unset LD_LIBRARY_PATH` to re-link executables.) ### Docker From 759dcdcfdbb7c40ee7631d6e52aa59c174103c86 Mon Sep 17 00:00:00 2001 From: JackJollimore <130917767+JackJollimore@users.noreply.github.com> Date: Fri, 1 Sep 2023 03:19:39 -0300 Subject: [PATCH 18/18] fix typo --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 14687c2719d68..2233078556204 100644 --- a/README.md +++ b/README.md @@ -835,11 +835,11 @@ make LLAMA_CLBLAST=1 Use one of the following to enable GPU: ``` -export LD_LIBRARY_PATH=/vendor/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=/vendor/lib64 ``` or ``` -export LD_LIBRARY_PATH=/system/vendor/lib64:$LD_LIBRARY_PATH +export LD_LIBRARY_PATH=/system/vendor/lib64 ``` then `./main ... --gpu-layers 1`