Self-Hosting Open WebUI & Ollama: The Production Docker Guide

Introduction

By 2026 the case for a private model server no longer rests on ideology. It rests on invoices and on audit trails. A team of fifteen developers pasting stack traces into a hosted chat product generates a bill that scales with usage, a retention policy nobody on the team wrote, and a compliance question the first time a customer asks where their data went. Meanwhile, open-weight models in the 8B to 32B range now handle summarization, code review, and document question answering well enough that the quality gap has stopped being the deciding factor for most internal workloads.

The friction has moved. Running a model is easy; running a model service is the hard part. You need a chat interface that non-engineers will actually use, authentication so the whole office is not an open endpoint, retrieval so people can ask questions about their own documents, GPU scheduling so one long generation does not freeze everyone else, and persistent storage so a container restart does not erase every conversation.

This guide covers the combination that has become the default answer: Ollama as the inference runtime and Open WebUI as the interface, both under Docker Compose. We go through how the two processes divide responsibility, how the retrieval pipeline works under the hood, how this stack compares with alternatives such as PrivateGPT and a raw llama.cpp server, and a complete, production-oriented Compose file with the environment variables that matter. If you search for "ollama webui" you will find plenty of five-line quickstarts. This one is for the version you leave running.

Architectural Breakdown & Core Mechanics

Two processes, two jobs

Ollama and Open WebUI are separate services with a clean boundary, and understanding that boundary is the key to debugging almost every problem you will hit.

Ollama owns models. It downloads GGUF-format weights from its registry, keeps them in a local store, loads them into GPU or CPU memory on demand, and serves an HTTP API on port 11434. It handles quantization variants, context window configuration, and model lifecycle (load, keep warm, unload). It has no opinion about users, conversations, or documents.

Open WebUI owns everything above that line: accounts and roles, chat history, system prompts, model presets, the document upload and retrieval pipeline, tool calling, and the browser UI itself. It stores state in a data directory (SQLite by default, Postgres optionally) and talks to Ollama, or to any OpenAI-compatible endpoint, over HTTP.

Because they only meet at an HTTP boundary, you can scale or replace either side independently. Swap Ollama for vLLM when you need higher throughput on a multi-GPU box, and Open WebUI keeps working because vLLM exposes an OpenAI-compatible API. Point Open WebUI at a remote Ollama on a GPU workstation while the UI runs on a cheap VM. The Compose file below keeps both on one host, but the seams are real.

How document RAG works in Open WebUI

When a user uploads a PDF or attaches a knowledge collection, Open WebUI runs a standard retrieval pipeline:

1. Extraction. The file is parsed to text (the built-in loaders handle PDF, DOCX, plain text, and more; an external extraction engine can be configured for scanned documents).

1. Chunking. Text is split into overlapping chunks. Chunk size and overlap are configurable and matter more than most people expect: too large and retrieval pulls irrelevant context, too small and answers lose surrounding meaning.

1. Embedding. Each chunk is turned into a vector by an embedding model. By default this is a small sentence-transformer running inside the Open WebUI container. You can instead route embeddings through Ollama and use a dedicated model such as nomic-embed-text, or pick a multilingual one like BGE-M3.

1. Storage. Vectors go into a vector database. Chroma is the default and lives on disk inside the data volume. Postgres with pgvector is supported and is the better choice once more than a handful of people share collections.

1. Retrieval and generation. At query time the question is embedded, the nearest chunks are fetched (optionally reranked), and they are injected into the prompt that goes to the chat model.

Two consequences matter operationally. The embedding model must stay constant for a given collection; change it and you must re-index, because vectors from different models are not comparable. And retrieval quality is bounded by the chat model's effective context length, so a document chat that works with a 32k-context model may truncate silently on a model loaded at Ollama's smaller default context.

Comparative Benchmarks & Evaluation Matrix

The table compares the realistic options for a private chat-plus-documents deployment. Ratings are qualitative and reflect typical configurations rather than a single controlled benchmark, since throughput depends heavily on GPU, quantization, and context length. Measure on your own hardware before committing.

The practical reading: start with Open WebUI and Ollama. Move the inference side to vLLM only when queueing becomes visible to users, and keep the UI.

Practical Step-by-Step Implementation Recipe

Prerequisites

You need Docker Engine with the Compose plugin and, for GPU acceleration, an NVIDIA driver plus the NVIDIA Container Toolkit. Verify the toolkit before anything else:

docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smi

If that prints your GPU table, containers can see the device. If it fails, fix it now; every later symptom ("model runs on CPU", "out of memory") traces back to this.

The Compose file

Create a project directory with a .env file holding secrets, then the Compose file. Generate the secret with openssl rand -hex 32 rather than inventing one.

mkdir -p ~/private-ai && cd ~/private-ai

printf 'WEBUI_SECRET_KEY=%s\n' "$(openssl rand -hex 32)" > .env

chmod 600 .env

# docker-compose.yml

services:

ollama:

image: ollama/ollama:latest

container_name: ollama

restart: unless-stopped

environment:
- OLLAMA_KEEP_ALIVE=30m          # keep models warm between requests
- OLLAMA_NUM_PARALLEL=2          # parallel requests per loaded model
- OLLAMA_MAX_LOADED_MODELS=2     # chat model + embedding model
- OLLAMA_FLASH_ATTENTION=1       # lower KV-cache memory on supported GPUs

volumes:
- ollama_data:/root/.ollama

deploy:

resources:

reservations:

devices:
- driver: nvidia

count: all

capabilities: [gpu]

healthcheck:

test: ["CMD", "ollama", "list"]

interval: 30s

timeout: 10s

retries: 5

# No published ports: only Open WebUI needs to reach Ollama.

open-webui:

image: ghcr.io/open-webui/open-webui:main

container_name: open-webui

restart: unless-stopped

depends_on:

ollama:

condition: service_healthy

ports:
- "127.0.0.1:3000:8080"          # bind to loopback; put a reverse proxy in front

environment:
- OLLAMA_BASE_URL=http://ollama:11434
- WEBUI_SECRET_KEY=${WEBUI_SECRET_KEY}
- ENABLE_SIGNUP=false            # admins create accounts
- DEFAULT_USER_ROLE=pending

# RAG: route embeddings through Ollama with a dedicated model
- RAG_EMBEDDING_ENGINE=ollama
- RAG_EMBEDDING_MODEL=nomic-embed-text
- CHUNK_SIZE=1000
- CHUNK_OVERLAP=100

volumes:
- open_webui_data:/app/backend/data

volumes:

ollama_data:

open_webui_data:

A few choices here deserve explanation. Ollama publishes no ports, so it is unreachable from the network; only the Open WebUI container can talk to it over the Compose network. Open WebUI binds to 127.0.0.1, which forces you to terminate TLS in a reverse proxy (Caddy and nginx are both fine) instead of exposing plain HTTP. ENABLE_SIGNUP=false plus DEFAULT_USER_ROLE=pending means nobody who discovers the URL can create a working account. Keep in mind that the first account created on a fresh instance becomes the administrator, so create yours immediately after first start, before the proxy is public.

Bring it up and pull models

docker compose up -d

docker compose logs -f open-webui   # wait for "Uvicorn running"

# Pull a chat model and the embedding model

docker exec ollama ollama pull qwen3:8b

docker exec ollama ollama pull nomic-embed-text

# Confirm the model is on the GPU, not the CPU

docker exec ollama ollama run qwen3:8b "Say ready." >/dev/null

docker exec ollama ollama ps

The ollama ps output has a processor column. You want it to report GPU. A split such as "40%/60% CPU/GPU" means the model plus its context does not fit in VRAM; choose a smaller quantization, a smaller model, or reduce context.

Set the context window deliberately

Ollama's default context is deliberately conservative, and RAG prompts with five retrieved chunks can exceed it, causing silent truncation. Set num_ctx per model in Open WebUI under the model's advanced parameters, or bake it into a derived model:

cat > Modelfile <<'EOT'

FROM qwen3:8b

PARAMETER num_ctx 16384

PARAMETER temperature 0.3

SYSTEM "You answer using the provided context. If the context lacks the answer, say so."

EOT

docker exec -i ollama ollama create qwen3-rag:8b -f - < Modelfile

Raise num_ctx in steps and watch ollama ps for VRAM growth; the KV cache scales with context length, and the jump from 8k to 32k can push a model off the GPU.

Back up and update

State lives in two volumes. Back both up on a schedule, and update by pulling new images and recreating:

docker run --rm -v open_webui_data:/data -v "$PWD":/backup alpine \

tar czf /backup/open-webui-$(date +%F).tgz -C /data .

docker compose pull && docker compose up -d

Pin image tags to a specific version once the deployment is stable. :main and :latest are convenient for a first install and risky for a service people depend on.

Strategic Catalog Integrations

A running server is a platform, and the rest of the work is deciding what goes onto it. The AIFuller catalog covers most of that decision space.

Models. Ollama is the runtime entry in our directory, and the model library behind it is where you will spend selection time. For general chat and reasoning, Qwen3 8B is a strong default at a size that fits consumer GPUs. If code review and algorithmic work dominate, compare against DeepSeek models, and keep Llama 3.1 8B Instruct as a well-supported baseline. For retrieval, the two embedding options discussed above are nomic-embed-text for English-heavy corpora and BGE-M3 when documents span languages.

Voice. Self-hosting Whisper Large v3 keeps meeting audio inside your network.

Adjacent projects. If your needs outgrow a chat UI, our project directory lists the next layer up. LangChain and LlamaIndex are the usual frameworks for custom retrieval pipelines that call your Ollama endpoint, and vLLM is the throughput-oriented backend discussed in the matrix.

Prompts and rules. A private model is only as useful as the instructions it receives. Seed your shared presets with the Executive Summarizer for long-document digests, the Senior Code Reviewer for pull request triage, and the Zero-Shot Chain-of-Thought Protocol for reasoning-heavy questions. If you manage this stack as infrastructure code, the Docker & Kubernetes DevOps rules give your coding assistant sensible defaults when it edits Compose files, and the LangChain & CrewAI agentic rules apply when you build agents against the endpoint.

Learning. For engineers building on top of this, the Full Stack LLM Bootcamp covers the application layer that sits above the server you just deployed.

Frequently Asked Questions (FAQ)

How much VRAM do I need for Open WebUI with Ollama?

It depends on model size, quantization, and context. As a rough guide, a 4-bit 7B or 8B model fits in 8 GB of VRAM with a modest context, while 14B to 32B models want 16 to 24 GB. Remember the embedding model and the KV cache also consume memory. Use ollama ps to verify that the loaded model reports GPU placement, because a partial CPU split makes responses much slower.

Why does Open WebUI say it cannot connect to Ollama?

In Docker, localhost inside the Open WebUI container refers to that container, not the host. Use the Compose service name (http://ollama:11434) when both run in one Compose project, or http://host.docker.internal:11434 when Ollama runs on the host (add extra_hosts: ["host.docker.internal:host-gateway"] on Linux). Check the connection under Admin Settings, Connections.

How do I run DeepSeek locally in Open WebUI?

Pull a distilled DeepSeek reasoning model through Ollama with docker exec ollama ollama pull <model-tag>, using the tag shown in the Ollama library, and it appears in the Open WebUI model selector automatically. Size the choice to your VRAM; the full-size DeepSeek models are far beyond a single consumer GPU, so local setups use the smaller distilled variants.

Is it safe to expose Open WebUI to the internet?

Only behind TLS and with signup disabled, strong admin credentials, and ideally a VPN or identity-aware proxy in front. Never publish Ollama's port 11434 directly, since it has no authentication of its own.