Docker Basics: Containers, Compose & Swarm

📋 At a Glance

Docker packages software into containers — isolated units that run the same everywhere. For a local AI stack, Docker is the foundation: Ollama, Open WebUI, n8n, Grafana — everything runs in containers. Docker Compose orchestrates multiple containers on one machine, Docker Swarm distributes them across multiple machines.

What is Docker?

Docker is a platform that packages software into containers. A container includes everything the software needs — code, runtime, system libraries, configuration. Instead of installing "program X on operating system Y," you start a container that brings everything with it.

This solves the classic "it works on my machine" problem: containers run on your laptop the same way they run on a server.

Containers vs. Virtual Machines

Containers and VMs solve similar problems but at different levels. Containers are lighter, VMs provide stronger isolation.

Docker containers share the host kernel, VMs each have their own operating system.

Property	Container (Docker)	Virtual Machine
Startup Time	Seconds	Minutes
Size	MBs (image)	GBs (full OS)
Isolation	Process-level (shared kernel)	Hardware-level (own kernel)
Performance	Near native	Overhead from hypervisor
GPU Access	NVIDIA Container Toolkit	GPU Passthrough (IOMMU/PCIe)
Use Case	Services, microservices, AI stacks	Full OS isolation, Windows on Linux

💡 When to use what?

For AI workloads: Docker containers for everything that needs to start fast and stay lightweight (Ollama, n8n, Grafana). VMs for hardware isolation (GPU passthrough to a dedicated AI VM) or when you need multiple operating systems. In practice: combine both — VMs for hosts, containers for services.

Docker Compose: Multiple Containers as a Stack

Docker Compose defines multi-container applications in a single YAML file. Instead of typing 5 separate docker run commands, you write a docker-compose.yml and start everything with one command.

Example: Minimal AI Stack

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    depends_on:
      - ollama

volumes:
  ollama-data:

Start and Stop

# Start stack (background)
docker compose up -d

# Check status
docker compose ps

# View logs
docker compose logs -f ollama

# Stop stack
docker compose down

# Stop stack AND delete volumes (CAUTION: data lost!)
docker compose down -v

⚠️ Volumes are your data

docker compose down -v deletes ALL volumes — all stored models, databases, configurations. Without -v, data is preserved and the next docker compose up reuses it.

Docker Swarm: Containers Across Multiple Machines

Docker Swarm turns multiple machines into a cluster. Services run on whichever node has capacity. If a node goes down, containers are automatically moved to other nodes.

Property	Docker Compose	Docker Swarm
Hosts	1 machine	Multiple machines (cluster)
High Availability	No	Yes (auto-failover)
Rolling Updates	No	Yes (zero-downtime)
Load Balancing	No (manual)	Yes (ingress mesh)
Secrets Management	Environment variables	Docker Secrets (encrypted)
Complexity	Low	Medium
Recommendation	Development, small setups	Production, multiple machines

ℹ️ Compose or Swarm?

Start with Docker Compose. If you only have one machine, that is perfectly sufficient. You only need Docker Swarm when you have multiple machines and want to run services with high availability. Compose files can be deployed as Swarm stacks with minor adjustments.

GPU Access in Docker

For AI workloads, containers need GPU access. The NVIDIA Container Toolkit makes GPUs available inside Docker.

Install NVIDIA Drivers

NVIDIA drivers must be installed on the host system. On Ubuntu: sudo apt install nvidia-driver-550

NVIDIA Container Toolkit

# Installation (Ubuntu/Debian)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Test

docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi

⚠️ GPU in Swarm

Docker Swarm has NO native GPU support for --gpus. Workaround: Set node labels (e.g., gpu=rtx3090) and pin services via placement constraints. Additionally, set the NVIDIA runtime as default in /etc/docker/daemon.json.

Practical Tips

Situation	Command	What it Does
Container not running	docker compose logs service-name	Shows error messages of the container
Disk full	docker system prune -a	Deletes unused images, containers, networks
Restart after update	docker compose pull && docker compose up -d	Pulls new images and restarts containers
Inspect container	docker exec -it container-name bash	Opens shell inside running container
Check resources	docker stats	Shows CPU, RAM, network per container

💡 Limit Logging

Docker containers can produce extremely large log files. Set a log limit per service in your docker-compose.yml: logging: options: max-size: "50m". Without this limit, we have seen 55 GB log files in practice.

Das Wichtigste

✓Containers are lighter than VMs — seconds to start instead of minutes, MBs instead of GBs.
✓Docker Compose defines multi-container stacks in a YAML file. One command starts everything.
✓Docker Swarm distributes containers across multiple machines with auto-failover. Only needed with 2+ nodes.
✓GPU in Docker requires the NVIDIA Container Toolkit. In Swarm: node labels + placement constraints.
✓Volumes are your data. docker compose down -v deletes EVERYTHING. Without -v, data is preserved.

Sources

Docker Documentation: Get Started — Official Docker introduction
Docker Compose Documentation — docker-compose.yml reference
Docker Swarm Mode — Swarm architecture and commands
NVIDIA Container Toolkit — GPU access in Docker containers