AI | | 7 min read

Run Ollama in Docker with Tailscale: Self-Hosted AI on Your Home Lab | IT HomeLab

This guide deploys Ollama as a Docker container with a Tailscale sidecar — giving you a self-hosted AI model runtime accessible from anywhere on your Tailscale network. No API costs, no subscriptions, no data leaving your network. The setup builds on the standard Docker home lab environment (Ubuntu VM on Proxmox with Portainer, Watchtower, and Tailscale) — if you’ve followed along with those videos, you already have everything you need.

🎥 Watch the Video Tutorial

💡Why Ollama in Docker Instead of a VM?

The previous Ollama guide ran Ollama directly on a dedicated Ubuntu VM. Running it as a Docker container on your existing Docker host is a lighter-weight alternative — no separate VM overhead, managed alongside your other containers with Portainer and Watchtower, and accessible over Tailscale using the same sidecar pattern as the rest of your stack. It’s also easier to connect to other Docker services. Hermes Agent and Open WebUI running on the same Docker host can reach Ollama over the internal Docker network rather than going via Tailscale — faster and simpler.
ℹ️Note: Hardware used: Dell Latitude 5411 — Ubuntu VM on Proxmox with 4 CPU cores, 12GB RAM, 512GB NVMe. No GPU. All inference runs on CPU. GitHub: compose.yml

🛠 What You’ll Need

  • The Docker home lab environment from the Docker Home Lab Setup guide — Portainer, Watchtower, and Tailscale already configured
  • Ollama Docker image — pulled automatically by Compose
  • A Tailscale auth key for the Ollama sidecar
  • Enough disk space for your chosen models — plan for 3–27 GB per model

📋Step-by-Step Setup

1. Folder structure

Following the standard convention, create the stack folder: 
mkdir -p /media/docker/scripts/ollama
cd /media/docker/scripts/ollama
Model data and config will live under:
  • /media/docker/configs/ollama/ — Ollama config and model storage
  • /media/docker/configs/tailscale/ollama/ — Tailscale state for the sidecar

2. Environment variables

Add the Ollama-specific variables to your .env file alongside the existing base variables. The Tailscale auth key goes in .env.secrets as usual: 
# .env additions for Ollama
OLLAMA_HOST=0.0.0.0:11434
OLLAMA_KEEP_ALIVE=5m
OLLAMA_NUM_PARALLEL=1
OLLAMA_NUM_GPU=0
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_CONTEXT_LENGTH=8192

3. Environment variable reference

Each variable controls how Ollama behaves on a CPU-only home lab system:
  • OLLAMA_HOST=0.0.0.0:11434 — listens on all interfaces so other services (Hermes, Open WebUI, n8n) can connect to it
  • OLLAMA_KEEP_ALIVE=5m — keeps the loaded model in memory for 5 minutes after the last request, avoiding a slow reload on the next query
  • OLLAMA_NUM_PARALLEL=1 — only run one inference at a time on CPU-only systems; parallel requests on CPU will saturate your cores and slow everything down
  • OLLAMA_NUM_GPU=0 — CPU-only mode. Change to 1 if you have a compatible NVIDIA GPU
  • OLLAMA_MAX_LOADED_MODELS=1 — cap the number of models held in memory. On a 12GB RAM VM, loading more than one or two models at once causes memory pressure
  • OLLAMA_CONTEXT_LENGTH=8192 — maximum token context window per conversation. Lower this to reduce RAM usage; raise it if you have headroom and need longer conversations

4. compose.yml

services:
  # =====================
  # Ollama
  # =====================
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_KEEP_ALIVE=${OLLAMA_KEEP_ALIVE:-5m}
      - OLLAMA_NUM_PARALLEL=${OLLAMA_NUM_PARALLEL:-1}
      - OLLAMA_MAX_LOADED_MODELS=${OLLAMA_MAX_LOADED_MODELS:-1}
      - OLLAMA_CONTEXT_LENGTH=${OLLAMA_CONTEXT_LENGTH:-8192}
    volumes:
      - ${CONFIG}/ollama:/root/.ollama
    network_mode: service:ts-ollama
    depends_on:
      - ts-ollama
    # Uncomment below if this host has an NVIDIA GPU passed through to Docker
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]
  # =====================
  # Ollama Tailscale Sidecar
  # =====================
  ts-ollama:
    image: tailscale/tailscale:latest
    container_name: ts-ollama
    hostname: ollama
    restart: unless-stopped
    labels:
      - "com.centurylinklabs.watchtower.enable=true"
    cap_add:
      - NET_ADMIN
      - NET_RAW
    volumes:
      - ${CONFIG}/tailscale/ollama:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun
    environment:
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_AUTHKEY=${TS_AUTHKEY}
      - TS_HOSTNAME=ollama
      - TS_ACCEPT_DNS=true
      - TS_USERSPACE=false
    ports:
      - "11434:11434"
    networks:
      ollama_net:
        ipv4_address: 172.20.15.10
networks:
  ollama_net:
    driver: bridge
    ipam:
      config:
        - subnet: 172.20.15.0/24
          gateway: 172.20.15.1

5. Bash aliases for quick Docker commands

Add these aliases to your ~/.bashrc to save typing the full Compose command every time: 
# Add to ~/.bashrc
alias docker-up='docker compose --env-file /media/docker/scripts/.env --env-file /media/docker/secrets/.env.secrets up -d'
alias docker-down='docker compose --env-file /media/docker/scripts/.env --env-file /media/docker/secrets/.env.secrets down'
alias docker-logs='docker compose --env-file /media/docker/scripts/.env --env-file /media/docker/secrets/.env.secrets logs -f'
source ~/.bashrc
💡Tip: With these aliases set, bringing up the Ollama stack from its folder is just docker-up. No more typing out the full env-file paths every time.

6. Deploy the stack

From the /media/docker/scripts/ollama folder: 
docker-up
Check the container started correctly: 
docker compose ps
docker logs ollama
Ollama takes a moment to initialise on first start. Look for a line confirming it’s listening on port 11434.

7. Approve the Tailscale device

Open your Tailscale admin console — ollama should appear as a new device. Approve it if auto-approve isn’t enabled and disable key expiry so it stays connected permanently.

8. Pull your first model

Exec into the Ollama container to pull a model: 
docker exec -it ollama ollama pull llama2
Or pull from outside the container using the Tailscale URL once it’s registered: 
curl http://ollama.your-tailnet.ts.net:11434/api/pull -d '{\"name\": \"llama2\"}'
⚠️Warning: Models are large files. Plan your storage before pulling. A Llama 2 pull is ~3.8 GB, Mistral is ~4.7 GB. The first pull takes time depending on your internet speed — be patient and don’t interrupt it.

9. Recommended models for CPU-only home labs

These four strike the best balance between capability and resource usage on a CPU-only 12GB VM:
  • llama2 (3.8 GB) — excellent all-round model, balanced performance across most tasks
  • mistral (4.7 GB) — fast and efficient with strong quality, good for general use
  • neural-chat (4.1 GB) — optimised for conversational use cases
  • qwen (2.2 GB) — smallest option, fastest to load, good starting point for low-RAM setups

10. Test the API endpoint

Confirm Ollama is reachable and responding via the Tailscale URL: 
curl http://ollama.your-tailnet.ts.net:11434
You should get: Ollama is running. Test a generation request: 
curl http://ollama.your-tailnet.ts.net:11434/api/generate \\
  -d '{\"model\": \"llama2\", \"prompt\": \"Explain Docker in one sentence.\", \"stream\": false}'

11. Model management

Useful commands for managing models once you have several pulled: 
# List installed models
docker exec ollama ollama list
# Pull a new model docker exec ollama ollama pull mistral # Delete a model to free space docker exec ollama ollama rm llama2 # Monitor resource usage docker stats ollama

12. Connect Ollama to other services

With Ollama running on Tailscale, other services in your home lab can connect to it using the Tailscale hostname:
  • Hermes Agent — point the model URL to http://ollama.your-tailnet.ts.net:11434 in Hermes config
  • Open WebUI — set OLLAMA_BASE_URL=http://ollama.your-tailnet.ts.net:11434 in the Open WebUI container environment
  • n8n — use the HTTP Request node pointing to http://ollama.your-tailnet.ts.net:11434/api/generate
  • Claude Code — configure the Ollama API endpoint for local model use
⚠️Warning: Monitor your Docker resource usage with docker stats. Ollama on CPU can consume all available cores during inference — if it’s competing with other containers on the same host, consider the dedicated VM approach from the earlier Ollama guide instead.

✅Conclusion

You now have Ollama running as a Docker container — managed by Portainer, kept up to date by Watchtower, and accessible securely from anywhere via Tailscale. Pull any model from the Ollama library and use it from Hermes Agent, Open WebUI, n8n, or directly via the API. No cloud dependency, no API costs, complete privacy. The Docker approach keeps everything consolidated on your existing Docker host rather than adding a separate VM per service — and the same Tailscale sidecar pattern that connects your other containers works identically here. Related guides: Docker Home Lab Setup · Open WebUI + n8n + Ollama Stack · Ollama on a Dedicated VM

📺Watch the full video guide here: https://youtu.be/T2GjoDNu8Xg If you found this helpful, like and subscribe to IT HomeLab Online on YouTube for more tutorials.

☕Support the channel: Patreon · Buy Me a Coffee

Enjoyed this guide?

Subscribe to the channel for more homelab builds, Raspberry Pi projects, and AI automation tutorials.

▶ Watch on YouTube

Related Posts

AI

Hermes Agent on Raspberry Pi Zero 2 W: Portable AI via USB Gadget Mode | IT HomeLab

July 11, 2026 AI

Hermes Agent on Raspberry Pi 5: Portable Self-Hosted AI Agent | IT HomeLab

June 19, 2026 AI

Hermes Agent on Proxmox: Self-Hosted AI Agent with Telegram Control | IT HomeLab

June 13, 2026