Ollama is a single program that downloads open language models, runs them on
your own CPU or GPU, and exposes them through a local API on
127.0.0.1:11434. It handles the awkward parts — quantised weights, GPU offloading,
context management, keeping a model warm between requests — so that
ollama run llama3.2 is the whole workflow.
Because it runs locally, it works offline, costs nothing per token, and no prompt or document you paste into it is sent anywhere.
Before you start
The only real constraint is memory. A model has to fit in RAM, or in VRAM if it is going on the GPU, alongside everything else running. These are rough figures for the usual 4-bit quantised builds — add a gigabyte or two for a long context:
| Model size | Memory needed | Realistic on |
|---|---|---|
| 1–2B parameters | ~2 GB | Any laptop, a Raspberry Pi 5 |
| 3–4B | ~3 GB | An 8 GB machine, comfortably |
| 7–8B | ~5–6 GB | 16 GB of RAM, or an 8 GB GPU |
| 13–14B | ~9–10 GB | A 12 GB GPU, or 32 GB of RAM |
| 30–34B | ~20 GB | A 24 GB GPU |
| 70B | ~40 GB+ | Two large GPUs, or patience |
Check what you have before downloading anything:
free -h # RAM
nvidia-smi # VRAM, if you have an NVIDIA card
df -h /usr # models are large; 20 GB free is a sane start
Install with the official script
This is the normal route on Debian and Ubuntu. It detects your architecture, installs the
binary, creates a dedicated ollama system user, writes a systemd unit and starts the
service:
curl -fsSL https://ollama.com/install.sh | sh
Piping a script into a shell runs it immediately. It is worth reading first — which you can do without executing anything:
curl -fsSL https://ollama.com/install.sh -o /tmp/ollama-install.sh
less /tmp/ollama-install.sh
sh /tmp/ollama-install.sh
Install manually from the archive
Use this when you want a pinned version, an air-gapped machine, or simply to know exactly what landed on disk. The archive is compressed with zstd:
sudo apt install zstd
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr
On an ARM machine, swap amd64 for arm64 in the URL. Then create the
service by hand:
sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)
Write /etc/systemd/system/ollama.service:
[Unit]
Description=Ollama Service
After=network-online.target
[Service]
ExecStart=/usr/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now ollama
Check that it is running
ollama -v
systemctl status ollama
curl http://localhost:11434/api/tags # lists the models you have
active (running) from systemd, and a JSON response from curl, means you are done
with the installation.
Run your first model
ollama run llama3.2
That downloads the model if it is not present, loads it, and drops you into a chat prompt.
Type /bye to leave. Other well-known starting points:
ollama run qwen2.5:7b # strong at code and reasoning for its size
ollama run deepseek-r1:8b # shows its reasoning before answering
ollama run llava # accepts images alongside text
ollama run nomic-embed-text # embeddings, for search and RAG
The full catalogue — thousands of models and sizes — is at
ollama.com/library. The :7b style suffix picks a size; leave it off and
you get the default tag for that model.
Everyday commands
| Command | What it does |
|---|---|
ollama list | Models on disk, with their sizes. |
ollama ps | What is loaded right now — and whether it is on the GPU. |
ollama pull <model> | Download or update a model without running it. |
ollama run <model> "prompt" | One-shot: answer and exit. |
ollama show <model> | Parameters, template, licence, context size. |
ollama rm <model> | Delete a model and free the disk space. |
ollama cp a b | Copy a model, usually before customising it. |
ollama serve | Run the server in the foreground (for debugging). |
Talk to it over the API
Everything the CLI does goes through a local HTTP API, which is what makes Ollama useful for scripts and applications:
curl http://localhost:11434/api/generate -d '{
"model": "llama3.2",
"prompt": "Explain what a swap file is, in two sentences.",
"stream": false
}'
There is also an OpenAI-compatible endpoint, so any library written for OpenAI works by pointing it at your machine instead:
# Python, using the official OpenAI client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
model="llama3.2",
messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)
Configuration
Settings are environment variables on the service. Edit the unit without touching the installed file:
sudo systemctl edit ollama.service
Add an [Service] section with the variables you want:
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MODELS=/srv/ollama/models"
sudo systemctl daemon-reload
sudo systemctl restart ollama
| Variable | Effect |
|---|---|
OLLAMA_HOST | Address and port to listen on. Default 127.0.0.1:11434. |
OLLAMA_MODELS | Where models are stored. Useful when the system disk is small. |
OLLAMA_KEEP_ALIVE | How long a model stays in memory after the last request. 30m, or -1 to never unload. |
OLLAMA_NUM_PARALLEL | Simultaneous requests per model. Each one costs memory. |
OLLAMA_MAX_LOADED_MODELS | How many different models may be resident at once. |
Do not set OLLAMA_HOST=0.0.0.0 on a machine you do not control.
Ollama has no authentication: anyone who can reach the port can run any model on your hardware
and read your prompts. Keep it on 127.0.0.1 and reach it over SSH port forwarding,
or put a reverse proxy with auth in front of it.
Using the GPU
Ollama uses an NVIDIA GPU automatically when the proprietary driver is present. If you have not installed one yet, follow the NVIDIA drivers guide first — Ollama needs the driver, not the full CUDA toolkit.
ollama ps
The PROCESSOR column is the honest answer to “is it using my GPU?”:
100% GPU is what you want. A split like 60/40 CPU/GPU means the model is
too large for your VRAM and part of it is running on the CPU — either pick a smaller tag or accept
the slower speed. 100% CPU means no GPU was found at all.
On AMD, install the ROCm package as well as the main archive:
curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tar.zst | sudo tar x -C /usr
Where the models live
| Installed how | Path |
|---|---|
Official script (service as the ollama user) | /usr/share/ollama/.ollama/models |
| Running as your own user | ~/.ollama/models |
Models are blobs on disk; back them up by copying the directory, and free space with
ollama rm rather than deleting files by hand.
Troubleshooting
| Symptom | Fix |
|---|---|
ollama: command not found | The binary is in /usr/local/bin. Open a new terminal, or check your PATH. |
| The service will not start | journalctl -u ollama -n 50 names the reason — usually a port already in use or a permissions problem on the model directory. |
| Port 11434 already in use | ss -tulpn | grep 11434 to find the process, or set OLLAMA_HOST to another port. |
| Answers arrive very slowly | ollama ps will say 100% CPU. Either install the GPU driver or use a smaller model. |
| The model unloads mid-conversation | It does not fit in memory. Drop to a smaller tag, or raise OLLAMA_KEEP_ALIVE only if RAM allows. |
| A download stalls at 99% | Usually disk space. df -h the models directory. |
tar: zstd: not found | sudo apt install zstd, then repeat the extract. |
Updating and removing
# update: re-run the installer
curl -fsSL https://ollama.com/install.sh | sh
# remove completely
sudo systemctl disable --now ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm "$(which ollama)"
sudo rm -rf /usr/lib/ollama /usr/share/ollama
sudo userdel ollama
Deleting /usr/share/ollama removes every downloaded model with it — that is usually
tens of gigabytes, and the reason an uninstall frees so much space.
Optional: a chat interface
The terminal is fine for testing, but a browser front end is more comfortable for daily use.
Open WebUI is the common choice; it talks to Ollama on the same machine and adds
conversation history, document upload and multiple users. Install it in a Python virtual
environment or a container following its own documentation, then point it at
http://localhost:11434.