Skip to content

Commit 892fd78

Browse files
maouwmaouw
committed
Updated to latest
1 parent 7a12da8 commit 892fd78

File tree

4 files changed

+144
-193
lines changed

4 files changed

+144
-193
lines changed

‎README.md

+67-30
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ This container provides a convenient way to run [ollama](https://github.com/olla
77
First, you'll need to log in to Hyak. If you've never set this up, go [here](https://uw-psych.github.io/compute_docs).
88

99
```bash
10-
ssh your-uw-netid@klone.hyak.uw.edu # Replace `your-uw-netid` with your UW NetID
10+
# Replace `your-uw-netid` with your UW NetID:
11+
ssh your-uw-netid@klone.hyak.uw.edu
1112
```
1213

1314
Then, you'll need to request a compute node. You can do this with the `salloc` command:
@@ -18,66 +19,102 @@ Then, you'll need to request a compute node. You can do this with the `salloc` c
1819
salloc --account escience --partition gpu-a40 --mem 64G -c 2 --time 1:00:00 --gpus 1
1920
```
2021

21-
One you're logged in to the compute node, you should set up your cache directories and Apptainer settings.
22+
### Building the container
2223

23-
👉 *If you're following this tutorial, **you should do this every time you're running ollama on Hyak!** This is because the default settings for Apptainer will use your home directory for caching, which will quickly fill up your home directory and cause your jobs to fail. If you are aware of this and have already set `APPTAINER_CACHEDIR`, you can remove the line that sets `APPTAINER_CACHEDIR`.*
24+
Next, you'll need to build the container. In this example, we'll build the container in a directory in your scratch space. You can change the path to wherever you'd like to build the container.
2425

2526
```bash
26-
# Do this in every session where you're running ollama on Hyak!
27+
mkdir -p "/gscratch/scrubbed/${USER}/ollama"
28+
cd "/gscratch/scrubbed/${USER}/ollama"
29+
git clone https://github.com/uw-psych/ollama-container
30+
cd ollama-container
31+
apptainer build ollama.sif Singularity
32+
```
33+
34+
#### Specifying a different version of `ollama`
35+
36+
By default, the container will install the latest version of `ollama`. If you want to install a specific version, you can specify the version with the `OLLAMA_VERSION` build argument. The most recent version tested with this container is `0.5.8`.
37+
38+
### Starting the `ollama` server
39+
40+
The model files that `ollama` uses are stored by default in your home directory. As these files can be quite large, it's a good idea to store them somewhere else. In this example, we'll store them in your scratch space.
2741

28-
# Set up cache directories:
29-
export APPTAINER_CACHEDIR="/gscratch/scrubbed/${USER}/.cache/apptainer"
42+
```bash
3043
export OLLAMA_MODELS="/gscratch/scrubbed/${USER}/ollama/models"
31-
mkdir -p "${APPTAINER_CACHEDIR}" "${OLLAMA_MODELS}"
44+
mkdir -p "${OLLAMA_MODELS}"
45+
```
46+
47+
You should run the command above every time you start a new server. If you want to run it automatically every time you log in, you can add it to your `.bashrc` file.
3248

33-
# Set up Apptainer:
34-
export APPTAINER_BIND=/gscratch APPTAINER_WRITABLE_TMPFS=1 APPTAINER_NV=1
49+
Next, you'll have to start the `ollama` server. You can set the port for the server with the `OLLAMA_PORT` environment variable or leave it unset to use a random port.
50+
51+
```bash
52+
# Start the ollama server as an Apptainer instance named "ollama-$USER":
53+
# --nv: Use the NVIDIA GPU
54+
# --writable-tmpfs: Use a writable tmpfs for the cache directory
55+
# --bind /gscratch: Bind /gscratch to the container
56+
apptainer instance start --nv --writable-tmpfs --bind /gscratch ollama.sif ollama-$USER
3557
```
3658

37-
Next, build the container. (You only need to do this once.)
59+
### Running `ollama` commands
60+
61+
To run `ollama` commands, execute the `apptainer run` command with your instance as the first argument and the `ollama` command as the second argument.
62+
63+
For example, to get help with the `ollama` command, run:
3864

3965
```bash
40-
mkdir -p "/gscratch/scrubbed/${USER}/ollama"
41-
cd "/gscratch/scrubbed/${USER}/ollama"
42-
git clone https://github.com/uw-psych/ollama-container
43-
cd ollama-container
44-
apptainer build ollama.sif Singularity
66+
apptainer run instance://ollama-$USER ollama help
4567
```
4668

47-
Next, you'll have to start the ollama server. Before you do this, you'll need to find an open port to use. You can do this with the `random-port` command embedded in this container:
69+
You can start an interactive prompt with the following command:
4870

4971
```bash
50-
export OLLAMA_PORT="$(apptainer run ollama.sif random-port)"
72+
apptainer run instance://ollama-$USER ollama run qwen:0.5b
5173
```
5274

75+
Or provide the prompt on the command line and return JSON output non-interactively:
76+
5377
```bash
54-
# Start the ollama server (make sure you include the `&` at the end to run it in the background):
55-
apptainer run ollama.sif serve &
78+
# NOTE: Not all models support JSON output
79+
# NOTE: Wrap the prompt in single quotes to avoid issues with special characters
80+
apptainer run instance://ollama-$USER ollama run qwen:0.5b --format=json --prompt 'Who are you?'
81+
```
5682

57-
# Wait a few seconds for the server to start up:
58-
sleep 5
83+
For other models, you can replace `qwen:0.5b` with the name of the model you want to use. You can find a list of available models [here](https://ollama.ai/library).
84+
85+
To show the models on the server, run:
86+
87+
```bash
88+
apptainer run instance://ollama-$USER ollama list
5989
```
6090

61-
Once the server is running, you can start an interactive prompt with the following command:
91+
To show the currently running models, run:
6292

6393
```bash
64-
# Start an interactive prompt with the qwen:0.5b model:
65-
apptainer run ollama.sif run qwen:0.5b
94+
apptainer run instance://ollama-$USER ollama ps
6695
```
6796

68-
For other models, you can replace `qwen:0.5b` with the name of the model you want to use. You can find a list of available models [here](https://ollama.ai/library).
97+
To stop the server, run:
6998

70-
To list available models, try:
99+
```bash
100+
apptainer instance stop ollama-$USER
101+
```
102+
103+
See the [documentation](https://github.com/ollama/ollama) for more information on how to use `ollama`.
104+
105+
#### Listing available models and tags
106+
107+
This repository includes a custom command to list available models and tags at (https://ollama.com/library). This command is not part of the `ollama` package and is only available in this container. It is useful for finding the names of models and tags to use with the `ollama` command, but it is not guaranteed to work in the future.
108+
109+
To list available models, try the following command with a running instance:
71110

72111
```bash
73-
apptainer run ollama.sif available
112+
apptainer run instance://ollama-$USER available-models
74113
```
75114

76115
To list available tags for a model, try:
77116

78117
```bash
79118
# Replace `qwen` with the name of the model you want to check:
80-
apptainer run ollama.sif available tags qwen
119+
apptainer run instance://ollama-$USER available-tags qwen
81120
```
82-
83-
See the [documentation](https://github.com/ollama/ollama) for more information on how to use `ollama`.

‎README.md.esh

-64
This file was deleted.

‎Singularity

+77-70
Original file line numberDiff line numberDiff line change
@@ -2,97 +2,104 @@ Bootstrap: docker
22
From: ollama/ollama:{{ OLLAMA_TAG }}
33

44
%arguments
5-
OLLAMA_TAG=0.5.8
5+
OLLAMA_TAG=latest
6+
OLLAMA_LIBRARY_URL=https://ollama.com/library
67

78
%setup
89
[ -n "${APPTAINER_ROOTFS:-}" ] && ./.build-scripts/write-apptainer-labels.sh >"${APPTAINER_ROOTFS}/.build_labels"
910

10-
%files
11-
ollama_available_models.sh /usr/local/bin/ollama-available-models
12-
1311
%post
14-
set -ex
15-
export DEBIAN_FRONTEND=noninteractive
16-
apt-get update -yq
17-
apt-get install -yq curl
18-
apt-get clean -yq
19-
rm -rf /var/lib/apt/lists/*
12+
if ! [ -f /etc/debian_version ]; then
13+
echo "This container is only fully supported on Debian-based systems. Some commands may not work." >&2
14+
else
15+
export DEBIAN_FRONTEND=noninteractive
16+
apt-get update -yq && \
17+
apt-get install -y curl && \
18+
apt-get clean &&
19+
rm -rf /var/lib/apt/lists/* /var/cache/apt/*
20+
fi
2021

2122
%test
2223
ollama --help
2324

2425
%runscript
25-
set -e
26-
get_random_port() {
27-
bash -c 'set -e; read LOWERPORT UPPERPORT < /proc/sys/net/ipv4/ip_local_port_range; PORTRANGE=$(( UPPERPORT - LOWERPORT )); while :; do PORT=$(( LOWERPORT + ( RANDOM % PORTRANGE) )); (echo -n > /dev/tcp/127.0.0/${PORT}) &>/dev/null || echo "${PORT}" && exit 0; done; exit 1' || {
28-
echo "Failed to find an open port. Exiting." >&2
26+
set -o noglob -o errexit
27+
if [ $# -eq 0 ] || [ -z "${1:-}" ]; then
28+
echo "Usage: $0 ollama [args]" >&2
29+
exit 1
30+
fi
31+
OLLAMA_LIBRARY_URL="${OLLAMA_LIBRARY_URL:-{{ OLLAMA_LIBRARY_URL }}}"
32+
case "${1:-}" in
33+
available-models)
34+
shift
35+
command -v curl >/dev/null || { echo "curl not found. Exiting." >&2; exit 1; }
36+
curl -sL "${OLLAMA_LIBRARY_URL}" | grep -oP 'href="/library/\K[^"]+' || { echo "No models found at ${OLLAMA_LIBRARY_URL}" >&2; exit 2; }
37+
exit 0
38+
;;
39+
available-tags)
40+
shift
41+
command -v curl >/dev/null || { echo "curl not found. Exiting." >&2; exit 1; }
42+
curl -sL "${OLLAMA_LIBRARY_URL}/$1/tags" | grep -o "$1:[^\" ]*q[^\" ]*" | grep -E -v 'text|base|fp|q[45]_[01]' || { echo "No model tags found at ${OLLAMA_LIBRARY_URL}/$1/tags" >&2; exit 2; }
43+
exit 0
44+
;;
45+
*) ;;
46+
esac
47+
# Run the ollama command with the remaining arguments:
48+
"$@"
49+
50+
%startscript
51+
set -o errexit -o noglob
52+
if [ -z "${OLLAMA_PORT:-}" ]; then
53+
echo "OLLAMA_PORT not set. Finding a random port to use" >&2
54+
get_random_port() {
55+
bash -c 'set -e; read LOWERPORT UPPERPORT < /proc/sys/net/ipv4/ip_local_port_range; PORTRANGE=$(( UPPERPORT - LOWERPORT )); while :; do PORT=$(( LOWERPORT + ( RANDOM % PORTRANGE) )); (echo -n > /dev/tcp/127.0.0/${PORT}) &>/dev/null || echo "${PORT}" && exit 0; done; exit 1'
56+
}
57+
OLLAMA_PORT="$(get_random_port)" || {
58+
echo "Failed to get random port. Exiting." >&2
2959
exit 1
3060
}
31-
}
32-
33-
if [ "${1:-}" = "random-port" ]; then
34-
get_random_port
35-
exit $?
3661
fi
37-
38-
# Set up OLLAMA_HOST if OLLAMA_PORT is set:
39-
export OLLAMA_HOST="${OLLAMA_HOST:-127.0.0.1${OLLAMA_PORT:+:${OLLAMA_PORT}}}"
40-
62+
4163
# If we're runninng on klone, we should have access to /gscratch/scrubbed.
4264
# However, by default, /gscratch is not mounted in the container, whereas /mmfs1 is.
4365
# /gscratch is the same as /mmfs1/gscratch, so we can use /mmfs1/gscratch/scrubbed.
44-
if [ -d "/gscratch/scrubbed" ]; then
45-
SCRUBBED_DIR="/gscratch/scrubbed/${USER}"
46-
elif [ -d "/mmfs1/gscratch/scrubbed" ]; then
47-
SCRUBBED_DIR="/mmfs1/gscratch/scrubbed/${USER}"
48-
case "${OLLAMA_MODELS:-}" in
49-
/gscratch/*)
50-
OLLAMA_MODELS="${OLLAMA_MODELS#/gscratch/}"
51-
OLLAMA_MODELS="/mmfs1/gscratch/${OLLAMA_MODELS}"
52-
;;
53-
*) ;;
54-
esac
66+
if findmnt -M /gscratch -O rw -f 2>/dev/null || findmnt -M /mmfs1 -O rw -f 2>/dev/null; then
67+
OLLAMA_MODELS="${OLLAMA_MODELS:-${USER_SCRUBBED_DIR:-/gscratch/scrubbed}/${USER}/ollama/models}}"
68+
OLLAMA_MODELS_REAL="$(realpath "${OLLAMA_MODELS}")"
69+
if ! mkdir -p "${OLLAMA_MODELS}" 2>/dev/null || ! { [ -d "${OLLAMA_MODELS}" ] && [ ! -w "${OLLAMA_MODELS}" ]; } && [ -d "/mmfs1" ]; then
70+
case "${OLLAMA_MODELS_REAL}" in
71+
/gscratch/*) OLLAMA_MODELS="/mmfs1${OLLAMA_MODELS_REAL}" ;;
72+
*) ;;
73+
esac
74+
fi
5575
fi
56-
[ -n "${SCRUBBED_DIR:-}" ] && OLLAMA_MODELS="${OLLAMA_MODELS:-${SCRUBBED_DIR}/ollama/models}"
57-
[ -n "${OLLAMA_MODELS:-}" ] && export OLLAMA_MODELS
58-
59-
# If no arguments are given, default to running `ollama serve`:
60-
[ $# -eq 0 ] && set -- serve
61-
62-
# If the first argument is "serve",
63-
# 1. create the models directory if it doesn't exist
64-
# 2. write a descriptive message to stderr
65-
if [ "${1:-}" = "serve" ]; then
66-
if [ -z "${OLLAMA_PORT:-}" ]; then
67-
echo "OLLAMA_PORT not set. Finding a random port to use" >&2
68-
OLLAMA_PORT="$(get_random_port)"
69-
if [ $? -ne 0 ]; then
70-
echo "Failed to get random port. Exiting." >&2
71-
exit 2
72-
fi
76+
if [ -n "${OLLAMA_MODELS:-}" ]; then
77+
if [ -f "${OLLAMA_MODELS}" ]; then
78+
echo "\"${OLLAMA_MODELS}\" is a file. Exiting." >&2
79+
exit 1
7380
fi
74-
75-
if [ -n "${OLLAMA_MODELS:-}" ]; then
76-
mkdir -p "${OLLAMA_MODELS}" || {
77-
echo "Failed to create the OLLAMA_MODELS=\"${OLLAMA_MODELS}\" directory. Exiting." >&2
78-
exit 1
79-
}
81+
if ! mkdir -p "${OLLAMA_MODELS}" 2>/dev/null; then
82+
echo "Failed to create the directory \"${OLLAMA_MODELS}\". Exiting." >&2
83+
exit 1
8084
fi
81-
echo "Starting ollama server${OLLAMA_HOST:+ on ${OLLAMA_HOST}}${OLLAMA_MODELS:+ with models in ${OLLAMA_MODELS}}." >&2
82-
fi
83-
84-
if [ "${1:-}" = "available" ]; then
85-
if ollama available 2>&1 | grep -qE 'Error:.*unknown command'; then
86-
shift
87-
ollama-available-models "$@"
88-
exit $?
85+
if ! [ -r "${OLLAMA_MODELS}" ]; then
86+
echo "The directory \"${OLLAMA_MODELS}\" is not readable. Exiting." >&2
87+
exit 1
8988
fi
89+
if ! [ -w "${OLLAMA_MODELS}" ]; then
90+
echo "The directory \"${OLLAMA_MODELS}\" is not writable. Exiting." >&2
91+
exit 1
92+
fi
93+
export OLLAMA_MODELS
9094
fi
91-
# Run the ollama command with the given arguments:
92-
ollama "$@"
93-
94-
%startscript
95-
/.run serve "$@"
95+
# Set up OLLAMA_HOST
96+
export OLLAMA_HOST="${OLLAMA_HOST:-127.0.0.1${OLLAMA_PORT:+:${OLLAMA_PORT}}}"
97+
ollama serve || {
98+
echo "Failed to start ollama. Exiting." >&2
99+
exit 1
100+
}
101+
OLLAMA_PID=$!
102+
trap 'kill -TERM $OLLAMA_PID' TERM INT
96103

97104
%help
98105
This is a simple container for running Ollama. For more information, visit https://ollama.ai.

0 commit comments

Comments
 (0)