vLLM vs Ollama: Choosing a Local LLM Inference Server in 2026

Introduction

Most teams that self-host a model start the same way. Someone runs ollama run on a laptop, the answers are good enough, and a week later a second developer wants the same endpoint. A month later there are twelve people, an internal agent firing parallel requests, and a latency graph that looks like a staircase. At that point the question "vLLM vs Ollama" stops being a preference and becomes a capacity-planning problem.

Both projects expose an HTTP API that speaks the OpenAI chat-completions dialect, so on the surface they look interchangeable. Underneath they make opposite bets. Ollama optimizes for the person at the keyboard: one command to pull a quantized model, sensible defaults, runs on a MacBook, unloads models when idle. vLLM optimizes for the scheduler: many concurrent sequences sharing one GPU, memory managed in fixed-size blocks, throughput measured in tokens per second across all users rather than for one.

The pain points developers report in 2026 are consistent. Ollama feels fast for one user and then queues requests once concurrency exceeds its parallel slots. vLLM demands a CUDA or ROCm GPU, a longer cold start, and explicit decisions about context length and memory fraction that Ollama makes silently. People also mix up the model formats: Ollama pulls GGUF files, vLLM loads Hugging Face safetensors, and the quantization that fits a 24 GB card is not the same artifact in each.

This article explains how each server schedules work, compares them on the dimensions that decide real deployments, and gives a runnable recipe that puts both behind the same OpenAI-compatible client so you can measure them on your own hardware rather than trusting anyone's leaderboard, including ours.

Architectural Breakdown & Core Mechanics

Ollama: a model manager wrapped around llama.cpp

Ollama is a Go daemon that manages a local model library and drives a llama.cpp based runner process for inference. It pulls GGUF weights from its registry, stores them under a models directory, and loads one into memory when the first request arrives. It keeps the model warm for a configurable period (OLLAMA_KEEP_ALIVE), then unloads it. Concurrency is bounded by OLLAMA_NUM_PARALLEL (slots per loaded model) and OLLAMA_MAX_LOADED_MODELS; requests beyond the slot count wait in a queue (OLLAMA_MAX_QUEUE).

The upside is portability. The same binary uses Metal on Apple Silicon, CUDA on NVIDIA, ROCm on supported AMD cards, and falls back to CPU, and GGUF quantizations (Q4_K_M, Q5_K_M, Q8_0) let a 30B-class mixture-of-experts model run in 24 GB or less. The cost is that each parallel slot carries its own slice of KV cache, so raising the slot count multiplies memory use.

vLLM: a scheduler built around the KV cache

vLLM is a Python and CUDA engine whose central idea is PagedAttention. Instead of reserving one contiguous KV-cache buffer per sequence sized for the maximum context, it allocates the cache in fixed-size blocks and maps each sequence to blocks through a table, the way an operating system maps virtual pages. Fragmentation drops, identical prompt prefixes can share blocks (prefix caching), and the engine can admit many more sequences into the same memory.

On top of that sits continuous batching. Rather than waiting for a batch to finish, the scheduler adds new requests and retires finished ones at every decoding step, so the GPU stays saturated while individual requests start and end at different times. Add tensor parallelism (--tensor-parallel-size) for multi-GPU nodes and support for AWQ, GPTQ and FP8 weights, and you have an engine tuned for serving many users from the same weights.

What this means in practice

For a single interactive user, per-token latency is dominated by memory bandwidth and both engines are in the same range; differences come from quantization and kernel quality more than from architecture. As concurrency rises, vLLM's block allocator and batching let aggregate throughput climb while Ollama's per-slot design flattens out and requests queue. That crossover point depends on your model, context length and GPU, which is why the recipe below measures it instead of asserting it.

Comparative Benchmarks & Evaluation Matrix

No throughput numbers appear below on purpose. Tokens per second varies by an order of magnitude with GPU generation, quantization, context length and batch composition, and any figure we printed would be an estimate about hardware we did not run. The matrix compares design properties; the script in the next section produces the measured numbers for your own machine.

A practical rule of thumb: if the answer to "how many requests are in flight at the same time" is usually one or two, Ollama's simplicity wins. If it is routinely ten or more, or if an agent framework fans out parallel tool calls, move the shared endpoint to vLLM and keep Ollama on developer machines.

Practical Step-by-Step Implementation Recipe

The recipe runs both servers on one NVIDIA host and benchmarks them with the same client. It assumes Docker with the NVIDIA Container Toolkit installed. Model IDs are examples; substitute whatever fits your VRAM, such as Qwen3 8B or the GPT-OSS 20B weights.

Step 1: Compose file

services:

ollama:

image: ollama/ollama:latest

ports:
- "11434:11434"

environment:

OLLAMA_NUM_PARALLEL: "4"

OLLAMA_MAX_LOADED_MODELS: "1"

OLLAMA_KEEP_ALIVE: "30m"

volumes:
- ollama-data:/root/.ollama

deploy:

resources:

reservations:

devices:
- driver: nvidia

count: all

capabilities: [gpu]

vllm:

image: vllm/vllm-openai:latest

ports:
- "8000:8000"

ipc: host

command: >

--model Qwen/Qwen3-8B

--max-model-len 8192

--gpu-memory-utilization 0.85

--max-num-seqs 64

--enable-prefix-caching

volumes:
- hf-cache:/root/.cache/huggingface

environment:

HUGGING_FACE_HUB_TOKEN: "${HF_TOKEN:-}"

deploy:

resources:

reservations:

devices:
- driver: nvidia

count: all

capabilities: [gpu]

volumes:

ollama-data:

hf-cache:

Running both on one GPU at the same time will not fit in most cards because vLLM reserves its memory fraction up front. Start one service, measure, stop it, then start the other: docker compose up -d ollama, then later docker compose up -d vllm.

Step 2: Pull the Ollama model

docker compose exec ollama ollama pull qwen3:8b

curl -s http://localhost:11434/v1/models | head -c 300

curl -s http://localhost:8000/v1/models | head -c 300   # when vllm is running

Step 3: Concurrency benchmark

The script sends N identical streaming requests at once and reports time to first token and total completion throughput. It uses only the Python standard library plus httpx.

import asyncio, json, sys, time

import httpx

BASE, MODEL, N = sys.argv[1], sys.argv[2], int(sys.argv[3])

PROMPT = "Explain how a B-tree handles node splits, in about 300 words."

async def one(client):

start, first, tokens = time.perf_counter(), None, 0

body = {"model": MODEL, "stream": True, "max_tokens": 300,

"messages": [{"role": "user", "content": PROMPT}]}

async with client.stream("POST", f"{BASE}/v1/chat/completions", json=body) as r:

async for line in r.aiter_lines():

if not line.startswith("data: ") or line.endswith("[DONE]"):

continue

delta = json.loads(line[6:])["choices"][0]["delta"].get("content")

if delta:

first = first or time.perf_counter() - start

tokens += 1

return first or 0.0, tokens, time.perf_counter() - start

async def main():

async with httpx.AsyncClient(timeout=300) as client:

t0 = time.perf_counter()

results = await asyncio.gather(*[one(client) for _ in range(N)])

wall = time.perf_counter() - t0

ttft = sorted(r[0] for r in results)

total = sum(r[1] for r in results)

print(f"concurrency={N} median_ttft={ttft[len(ttft)//2]:.2f}s "

f"aggregate_chunks_per_s={total/wall:.1f}")

asyncio.run(main())

Run it at several concurrency levels against each backend:

pip install httpx

for n in 1 4 16 32; do

python bench.py http://localhost:11434 qwen3:8b $n

done

# stop ollama, start vllm, then:

for n in 1 4 16 32; do

python bench.py http://localhost:8000 Qwen/Qwen3-8B $n

done

The script counts streamed content chunks, which approximate tokens but are not identical to them, so compare backends with the same script rather than against published figures. Watch two columns: median time to first token (where queuing shows up) and aggregate throughput (where batching shows up). The concurrency level where Ollama's time to first token starts climbing is your capacity ceiling for that configuration.

Step 4: Point your tools at the winner

Both servers accept an OpenAI-style base URL, so switching is a configuration change: http://host:11434/v1 for Ollama or http://host:8000/v1 for vLLM, with any placeholder API key. If you put vLLM behind a team endpoint, start it with --api-key and put TLS in front.

Strategic Catalog Integrations

The comparison above is easier to apply with the surrounding pieces in hand. The catalog lists vLLM as a tool, and the repositories are indexed as vLLM on GitHub and Ollama on GitHub. If you need the lowest-level control or an unusual device, the llama.cpp project is the engine underneath Ollama.

A chat front end turns either server into something non-engineers will use; Open WebUI connects to both through their OpenAI-compatible endpoints, and PrivateGPT adds document question answering on top. For model choices that fit consumer VRAM, start with Qwen3 Coder 30B A3B GGUF for code, which loads in Ollama and llama.cpp but needs a safetensors build for vLLM.

For the operational side, the Docker and Kubernetes DevOps rules cover non-root containers, health probes and resource limits that apply directly to the Compose file above, and the FastAPI and Pydantic v2 rules help when you wrap either server in a thin authenticated gateway. The HTTP Benchmarking Tool CLI prompt is a quick way to generate a load-testing variant of the benchmark script. To understand why batching and KV caches behave the way they do, the Generative AI with LLMs course covers inference-time memory and scaling.

Frequently Asked Questions (FAQ)

Is vLLM faster than Ollama?

For aggregate throughput under concurrent load, vLLM is designed to be faster because of continuous batching and paged KV-cache management. For a single user generating one response at a time, the gap is small and can go either way depending on quantization and kernels. Measure with the script above on your hardware, at the concurrency your workload actually has.

Can I run vLLM on a Mac or without a GPU?

vLLM targets NVIDIA CUDA and AMD ROCm GPUs. CPU and Apple Silicon support exists in limited or experimental forms and is not the main path. If your hardware is a Mac, Ollama or llama.cpp with Metal is the practical choice.

Can vLLM load the same GGUF files that Ollama uses?

vLLM's native format is Hugging Face safetensors, with AWQ, GPTQ and FP8 quantizations. GGUF loading has been experimental and limited in coverage, so plan on downloading a separate safetensors or AWQ build of the model rather than reusing your Ollama files.

How do I use both with the same IDE or agent?

Because both expose /v1/chat/completions, configure two OpenAI-compatible providers with different base URLs and model names. A common pattern is Ollama locally for offline work and a shared vLLM endpoint for agents and CI jobs, with the client choosing by environment variable.