llama.cpp vs Ollama: Which Local LLM Runtime Should You Run in 2026?

Introduction

Almost every local-LLM setup in 2026 runs on the same inference core. Ollama wraps it, LM Studio wraps it, and a long list of desktop apps wrap it. That core is llama.cpp, and the practical question for a developer is whether to use it directly through llama-server or let Ollama manage it.

The question comes up when something stops being convenient. A model that fits in memory under Ollama's defaults runs with a smaller context than expected. A new model architecture lands in llama.cpp and is not yet in the Ollama library. A team wants one box to serve four people, and request queueing matters. Or a laptop fan spins up because a model stayed loaded after the chat window closed. Each of these is a configuration problem that one of the two tools hides and the other exposes.

The trade is control against ceremony. Ollama gives you ollama run, a model registry, automatic loading and unloading, and an OpenAI-compatible API in one binary. llama.cpp gives you every flag: context size, GPU layer offload, KV-cache quantization, speculative decoding, grammar-constrained sampling, and the ability to run a GGUF file you downloaded from Hugging Face five minutes after it appears.

This article explains how the two relate, compares them across the settings that change real outcomes, and provides a script that starts both on the same GGUF model and measures time-to-first-token and decode speed on your own hardware. There are no throughput numbers in the text, because they depend on your GPU, quantization, and context length; the evaluation matrix is a set of design observations. If you serve many concurrent users from a GPU, read the separate comparison of vLLM and Ollama as well, since neither tool here is built around a batch scheduler.

Architectural Breakdown & Core Mechanics

llama.cpp: a C/C++ inference library with a server on top

llama.cpp is a C/C++ implementation of transformer inference built on the ggml tensor library. Its native model format is GGUF, a single file containing weights, tokenizer, and metadata. Quantization schemes (the Q4_K_M family, Q5, Q6, Q8, and newer IQ variants) reduce weight size so a model fits in the memory you have. Backends cover Metal on Apple Silicon, CUDA, HIP, Vulkan, SYCL, and CPU with SIMD kernels.

The tool most developers use is llama-server, an HTTP server exposing an OpenAI-compatible /v1/chat/completions endpoint plus a small web UI. Its key knobs map directly to hardware decisions:

• -m or -hf selects a local GGUF file or downloads one from Hugging Face.
• -c sets total context, which is divided among parallel slots.
• -np sets the number of parallel slots; each slot decodes one sequence, and the server batches across active slots.
• -ngl sets how many layers are offloaded to the GPU, so a model larger than VRAM can split across GPU and system memory.
• Flash attention and KV-cache quantization flags reduce cache memory at some quality cost for long contexts.
• --jinja applies the chat template embedded in the model file, which matters for tool calling.

Ollama: a model manager that drives llama.cpp-derived runners

Ollama is a Go daemon. It maintains a local library of models pulled by name and tag, stores them as layered blobs, and starts a runner process when a request arrives. A Modelfile lets you set base model, default parameters, system prompt, and template. The daemon decides how many layers to offload based on free VRAM, keeps the model resident for a configurable keep-alive period, and unloads it afterward. Settings such as parallel request count, maximum loaded models, flash attention, and KV-cache type are environment variables on the daemon.

Historically the runner was llama.cpp, and Ollama's own documentation and release notes describe where it has added its own engine for some model families. The practical consequence is that support for a brand-new architecture can arrive in llama.cpp first, and that Ollama's behavior for a given model can differ from llama-server with the same GGUF because defaults differ.

The defaults are the real difference

Both tools run the same class of kernels, so decode speed on the same quantized weights tends to be similar when settings match. The differences that change your experience are defaults: Ollama picks a context length and offload split for you and may choose conservatively, while llama-server does exactly what the flags say and fails loudly if memory is short. If you ever see different output from the "same" model, compare context size, sampling parameters, and chat template before blaming the engine.

Comparative Benchmarks & Evaluation Matrix

This matrix compares configuration and workflow properties. No tokens-per-second figures are included, because they vary with hardware, quantization, and context; the script in the next section produces numbers for your machine.

Who should not choose Ollama: anyone who needs to pin exact context, cache type, and offload for reproducible benchmarks, or who must run a model that has not yet been packaged. Who should not choose raw llama-server: a team that wants non-engineers to switch models from a UI without touching a terminal. A reasonable pattern is Ollama for exploration and a pinned llama-server (or a container) for the one model that backs a shared service.

If concurrency is the real requirement, the honest alternative is neither: a batch-oriented engine such as vLLM keeps a GPU busy across many sequences, at the price of needing a supported GPU and safetensors-format weights.

Step-by-Step Implementation Recipe

The script below starts both servers against the same quantized model and measures streaming time-to-first-token and approximate decode rate with curl. It assumes Linux or macOS, llama-server on your PATH, Ollama installed, and jq and bc available. Edit the three variables at the top to match your model; the Hugging Face repo and tag shown are an example, so verify the name before running.

#!/usr/bin/env bash

# bench-local-runtimes.sh -- compare llama-server and Ollama on one model.

# Results depend on your hardware, quantization, context size and prompt.

set -euo pipefail

HF_REPO="${HF_REPO:-unsloth/Qwen3-8B-GGUF:Q4_K_M}"   # llama-server -hf target

OLLAMA_MODEL="${OLLAMA_MODEL:-qwen3:8b}"              # closest Ollama tag

CTX="${CTX:-8192}"

PROMPT="Explain the difference between a mutex and a semaphore in five sentences."

cleanup() { [[ -n "${LLAMA_PID:-}" ]] && kill "$LLAMA_PID" 2>/dev/null || true; }

trap cleanup EXIT

measure() {  # $1 = base URL, $2 = model name

local url="$1" model="$2" start first end

start=$(date +%s.%N)

first=""

while IFS= read -r line; do

[[ "$line" == data:* && "$line" != "data: [DONE]" ]] || continue

if [[ -z "$first" ]]; then first=$(date +%s.%N); fi

done < <(curl -sN "$url/v1/chat/completions" \

-H 'Content-Type: application/json' \

-d "$(jq -n --arg m "$model" --arg p "$PROMPT" \

'{model:$m, stream:true, max_tokens:256, temperature:0,

messages:[{role:"user",content:$p}]}')")

end=$(date +%s.%N)

printf '  time-to-first-token: %.2fs   total: %.2fs\n' \

"$(echo "$first - $start" | bc -l)" "$(echo "$end - $start" | bc -l)"

}

echo "== llama-server (ctx=$CTX, all layers offloaded if they fit) =="

llama-server -hf "$HF_REPO" -c "$CTX" -ngl 99 --port 8080 --jinja >/tmp/llama-server.log 2>&1 &

LLAMA_PID=$!

until curl -sf http://127.0.0.1:8080/health >/dev/null; do sleep 1; done

measure "http://127.0.0.1:8080" "local"   # warm-up

measure "http://127.0.0.1:8080" "local"

kill "$LLAMA_PID"; wait "$LLAMA_PID" 2>/dev/null || true; unset LLAMA_PID

echo "== Ollama (Ollama defaults apply; set num_ctx to match for a fair test) =="

ollama pull "$OLLAMA_MODEL" >/dev/null

measure "http://127.0.0.1:11434" "$OLLAMA_MODEL"   # warm-up (includes model load)

measure "http://127.0.0.1:11434" "$OLLAMA_MODEL"

Interpret the output carefully. The first call to each server includes model loading, so only the second call reflects steady-state latency. The script counts streamed chunks rather than exact tokens, so use the total time as a relative comparison and read exact token rates from the timings field in the llama-server log or from ollama run --verbose. To compare fairly, align context size: set num_ctx in an Ollama Modelfile or request option to the same value passed to -c, and use the same quantization on both sides. Run each measurement several times, and record the GPU, driver, and software versions beside the result.

For a shared service, pin the configuration in a container so it is reproducible, and follow infrastructure conventions such as the Docker and Kubernetes DevOps rules for health checks, resource limits, and image pinning.

Strategic Catalog Integrations

These catalog entries connect directly to the runtime choice:

• Runtimes: read the llama.cpp project page and the Ollama project page, then keep Ollama in the tools directory as your reference for supported platforms.
• Batch-oriented alternative: if you outgrow slot-based serving, evaluate vLLM for GPU serving with continuous batching.
• Front end: put Open WebUI in front of either backend for chat, document upload, and multi-user accounts.
• GGUF models to test: try the Unsloth Qwen3 Coder 30B A3B GGUF for coding on a 24 GB-class machine, the Unsloth Qwen3 27B GGUF for general work, and gpt-oss-20b as an open-weight reference point.
• Small baseline: use Qwen3 8B as the model for the benchmark script on modest hardware.
• Ops hygiene: apply the Docker and Kubernetes DevOps rules when you containerize the pinned server.
• Learning path: the Full Stack LLM Bootcamp covers serving and evaluation trade-offs, and the beginner's guide to building and deploying LLMs prompt is a quick checklist for planning deployment.

Frequently Asked Questions (FAQ)

Is Ollama just llama.cpp?

Ollama began as a wrapper around llama.cpp and still relies on llama.cpp-derived code for many models, but it is a separate product with its own model registry, daemon, Modelfile format, and defaults. Its release notes describe where it uses its own engine. Treat them as related, not identical, and compare settings before comparing speed.

Is llama.cpp faster than Ollama?

On the same hardware, the same GGUF quantization, and matching context and offload settings, decode speed is usually close, because both are limited by memory bandwidth. Differences reported in practice often come from defaults: context length, how many layers are offloaded, flash attention, and KV-cache type. Measure with the script above on your own machine rather than relying on a general claim.

Can I use my own GGUF file with Ollama?

Yes. Create a Modelfile with a FROM line pointing at the GGUF path, add any PARAMETER and TEMPLATE settings the model needs, and run ollama create. Getting the chat template right is the usual stumbling block, and llama-server --jinja uses the template stored in the GGUF, which is a quick way to check what the model expects.

Which one should I use with Open WebUI or Cursor?

Both expose OpenAI-compatible endpoints, so either works. Ollama is the lower-friction choice for Open WebUI on a workstation because model switching is built in. A pinned llama-server is the better choice when you want a fixed model, fixed context, and a process you can supervise, for example as the backend for an editor assistant shared across a team.