Sangharsh OS

Documentation/Applications

Ollama & local AI

Ollama runs large language models on your own hardware — no account, no API key, and nothing you type leaves the machine. This guide covers installing it on a Debian-based system, choosing a model that fits your RAM, and getting it to use your GPU.

Ollama is a single program that downloads open language models, runs them on your own CPU or GPU, and exposes them through a local API on 127.0.0.1:11434. It handles the awkward parts — quantised weights, GPU offloading, context management, keeping a model warm between requests — so that ollama run llama3.2 is the whole workflow.

Because it runs locally, it works offline, costs nothing per token, and no prompt or document you paste into it is sent anywhere.

Before you start

The only real constraint is memory. A model has to fit in RAM, or in VRAM if it is going on the GPU, alongside everything else running. These are rough figures for the usual 4-bit quantised builds — add a gigabyte or two for a long context:

Model sizeMemory neededRealistic on
1–2B parameters~2 GBAny laptop, a Raspberry Pi 5
3–4B~3 GBAn 8 GB machine, comfortably
7–8B~5–6 GB16 GB of RAM, or an 8 GB GPU
13–14B~9–10 GBA 12 GB GPU, or 32 GB of RAM
30–34B~20 GBA 24 GB GPU
70B~40 GB+Two large GPUs, or patience

Check what you have before downloading anything:

free -h            # RAM
nvidia-smi         # VRAM, if you have an NVIDIA card
df -h /usr         # models are large; 20 GB free is a sane start

Install with the official script

This is the normal route on Debian and Ubuntu. It detects your architecture, installs the binary, creates a dedicated ollama system user, writes a systemd unit and starts the service:

curl -fsSL https://ollama.com/install.sh | sh

Piping a script into a shell runs it immediately. It is worth reading first — which you can do without executing anything:

curl -fsSL https://ollama.com/install.sh -o /tmp/ollama-install.sh
less /tmp/ollama-install.sh
sh /tmp/ollama-install.sh

Install manually from the archive

Use this when you want a pinned version, an air-gapped machine, or simply to know exactly what landed on disk. The archive is compressed with zstd:

sudo apt install zstd
curl -fsSL https://ollama.com/download/ollama-linux-amd64.tar.zst | sudo tar x -C /usr

On an ARM machine, swap amd64 for arm64 in the URL. Then create the service by hand:

sudo useradd -r -s /bin/false -U -m -d /usr/share/ollama ollama
sudo usermod -a -G ollama $(whoami)

Write /etc/systemd/system/ollama.service:

[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now ollama

Check that it is running

ollama -v
systemctl status ollama
curl http://localhost:11434/api/tags   # lists the models you have

active (running) from systemd, and a JSON response from curl, means you are done with the installation.

Run your first model

ollama run llama3.2

That downloads the model if it is not present, loads it, and drops you into a chat prompt. Type /bye to leave. Other well-known starting points:

ollama run qwen2.5:7b        # strong at code and reasoning for its size
ollama run deepseek-r1:8b    # shows its reasoning before answering
ollama run llava             # accepts images alongside text
ollama run nomic-embed-text  # embeddings, for search and RAG

The full catalogue — thousands of models and sizes — is at ollama.com/library. The :7b style suffix picks a size; leave it off and you get the default tag for that model.

Everyday commands

CommandWhat it does
ollama listModels on disk, with their sizes.
ollama psWhat is loaded right now — and whether it is on the GPU.
ollama pull <model>Download or update a model without running it.
ollama run <model> "prompt"One-shot: answer and exit.
ollama show <model>Parameters, template, licence, context size.
ollama rm <model>Delete a model and free the disk space.
ollama cp a bCopy a model, usually before customising it.
ollama serveRun the server in the foreground (for debugging).

Talk to it over the API

Everything the CLI does goes through a local HTTP API, which is what makes Ollama useful for scripts and applications:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain what a swap file is, in two sentences.",
  "stream": false
}'

There is also an OpenAI-compatible endpoint, so any library written for OpenAI works by pointing it at your machine instead:

# Python, using the official OpenAI client
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

Configuration

Settings are environment variables on the service. Edit the unit without touching the installed file:

sudo systemctl edit ollama.service

Add an [Service] section with the variables you want:

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_MODELS=/srv/ollama/models"
sudo systemctl daemon-reload
sudo systemctl restart ollama
VariableEffect
OLLAMA_HOSTAddress and port to listen on. Default 127.0.0.1:11434.
OLLAMA_MODELSWhere models are stored. Useful when the system disk is small.
OLLAMA_KEEP_ALIVEHow long a model stays in memory after the last request. 30m, or -1 to never unload.
OLLAMA_NUM_PARALLELSimultaneous requests per model. Each one costs memory.
OLLAMA_MAX_LOADED_MODELSHow many different models may be resident at once.

Do not set OLLAMA_HOST=0.0.0.0 on a machine you do not control. Ollama has no authentication: anyone who can reach the port can run any model on your hardware and read your prompts. Keep it on 127.0.0.1 and reach it over SSH port forwarding, or put a reverse proxy with auth in front of it.

Using the GPU

Ollama uses an NVIDIA GPU automatically when the proprietary driver is present. If you have not installed one yet, follow the NVIDIA drivers guide first — Ollama needs the driver, not the full CUDA toolkit.

ollama ps

The PROCESSOR column is the honest answer to “is it using my GPU?”: 100% GPU is what you want. A split like 60/40 CPU/GPU means the model is too large for your VRAM and part of it is running on the CPU — either pick a smaller tag or accept the slower speed. 100% CPU means no GPU was found at all.

On AMD, install the ROCm package as well as the main archive:

curl -fsSL https://ollama.com/download/ollama-linux-amd64-rocm.tar.zst | sudo tar x -C /usr

Where the models live

Installed howPath
Official script (service as the ollama user)/usr/share/ollama/.ollama/models
Running as your own user~/.ollama/models

Models are blobs on disk; back them up by copying the directory, and free space with ollama rm rather than deleting files by hand.

Troubleshooting

SymptomFix
ollama: command not foundThe binary is in /usr/local/bin. Open a new terminal, or check your PATH.
The service will not startjournalctl -u ollama -n 50 names the reason — usually a port already in use or a permissions problem on the model directory.
Port 11434 already in usess -tulpn | grep 11434 to find the process, or set OLLAMA_HOST to another port.
Answers arrive very slowlyollama ps will say 100% CPU. Either install the GPU driver or use a smaller model.
The model unloads mid-conversationIt does not fit in memory. Drop to a smaller tag, or raise OLLAMA_KEEP_ALIVE only if RAM allows.
A download stalls at 99%Usually disk space. df -h the models directory.
tar: zstd: not foundsudo apt install zstd, then repeat the extract.

Updating and removing

# update: re-run the installer
curl -fsSL https://ollama.com/install.sh | sh

# remove completely
sudo systemctl disable --now ollama
sudo rm /etc/systemd/system/ollama.service
sudo rm "$(which ollama)"
sudo rm -rf /usr/lib/ollama /usr/share/ollama
sudo userdel ollama

Deleting /usr/share/ollama removes every downloaded model with it — that is usually tens of gigabytes, and the reason an uninstall frees so much space.

Optional: a chat interface

The terminal is fine for testing, but a browser front end is more comfortable for daily use. Open WebUI is the common choice; it talks to Ollama on the same machine and adds conversation history, document upload and multiple users. Install it in a Python virtual environment or a container following its own documentation, then point it at http://localhost:11434.