An Ollama-like CLI experience for running LLMs with OpenVINO on Intel hardware.
Run local models through OpenAI-compatible and Ollama-compatible APIs, with a separate model converter and a lightweight inference container. Apache-2.0; model weights retain their own licenses.
Current source release: 2.6.0, including the October 18 benchmark/setup updates and October 19 publication maintenance. Stable runtime and optional nightly profiles remain separately pinned; see CHANGELOG.md.
| Start here | Contents |
|---|---|
| Arc Pro B50 quickstart | Build, select the correct GPU, configure, infer, and roll back |
| October 18, 2026 measurements | Three models, exact nightly versions, reproducible protocol and limitations |
| Security policy | Credential handling, trusted-network scope and reporting |
| OpenWebUI integration | Bring Illama, ComfyUI and local TTS/STT together with persistent chat/RAG |
This repository covers Illama LLM inference only. Image-generation setup lives in comfyui-intel-arc, and speech setup lives in speech-intel-arc.
The default build uses OpenVINO 2026.4.1. The measured 2026.5 nightly is an explicit optional profile, not a silent change to the stable builder or runner. Verified GPU inference does not prove that every model architecture can be exported or that every context fits in VRAM.
- Current architecture: decoupled
builder+runner+illamaCLI, with no active OVMS dependency. - Runtime pins: OpenVINO 2026.4.1, GenAI 2026.4.1.0, optimum-intel 2.2.0. Arc Pro B50 notes: docs/b50-model-notes.md.
- Completed in the current refactor:
config/server.jsonis the active server config surface- tokenizer-aware chat prompt handling
- real SSE streaming for
stream=true - exact pinned dependency versions in both Dockerfiles
- CLI health polling and stale OVMS cleanup
- OpenAI-style error objects and request limit knobs
- Runtime validation completed on the Ubuntu host:
- rebuilt images successfully
- verified
illama serve,illama ps,/illama/health, and/v1paths - verified OpenWebUI-compatible OpenAI API behavior
- confirmed existing downloaded models are reused from
/opt/illama/models
- 2026-05 hardening pass:
- GPU model cache wired through to the OpenVINO pipeline via
CACHE_DIR=$ILLAMA_CACHE_DIR(the previousOV_GPU_CACHE_MODEL=1env var was a no-op) - dependency bumps: openvino 2026.1.0, openvino-genai 2026.1.0.0, transformers 4.57.6, accelerate 1.13.0, fastapi 0.136.1, uvicorn 0.46.0, pydantic 2.13.3
- timing-safe bearer-token comparison via
secrets.compare_digest - bounded SSE event queue and tightened Prometheus label escaping
- CLI no longer splices unescaped values into Python source or curl
-dJSON bodies, andservecleans up stale exited containers before starting
- GPU model cache wired through to the OpenVINO pipeline via
- 2026-05 v2.2 additions:
- Ollama-compatible API (
/api/version,/api/tags,/api/chat,/api/generate,/api/show,/api/ps) — drop-in for Continue.dev, LangChainChatOllama, OpenWebUI's Ollama backend, etc. - OpenAI:
seed,response_format=json_object, andfrequency_penalty → repetition_penaltymapping - Stats persistence to
${ILLAMA_CACHE_DIR}/illama_stats.jsonso counters survive restarts - Docker
HEALTHCHECKreports an unhealthy runner; the restart policy handles process exits. Restarting an unhealthy but still-running process requires operator/orchestrator action. - Lazy
illama-builderimage —setup.shonly builds the runner; the conversion image is built on first non-OpenVINOillama pull - New CLI commands:
illama logs [N],illama bench,illama prune builder - Removed
bench.shin favour ofillama bench→scripts/benchmark_api.py
- Ollama-compatible API (
Two supported entry points run the same container with the same volumes and env: pick whichever fits your ops model.
| Path | When to use it | Files |
|---|---|---|
illama serve (bash CLI) |
Single host, manual control, illama symlinked into /usr/local/bin |
none beyond the install |
docker compose up -d |
Compose-based stacks, GitOps, CI/CD, pinning to a specific image tag | deploy/docker-compose.yml, deploy/.env |
systemctl start illama-runner |
Auto-start on boot, journalctl logs, layered on top of compose | deploy/illama-runner.service |
All three are interchangeable — illama stop + docker compose up -d (or vice-versa) works fine as long as the container name (illama-service) and host port match.
# 1. System packages
sudo apt update
sudo apt install -y docker.io docker-compose-plugin python3 curl
# 2. Docker group (so you don't need sudo for `docker ...`)
sudo usermod -aG docker "$USER"
# log out / log back in (or run `newgrp docker`) so it takes effect
# 3. Confirm Docker is usable
docker info >/dev/null && echo "docker OK"
docker compose version # need v2 plugin (Ubuntu 22.04+ ships it)For Intel GPU acceleration (any iGPU/dGPU with OpenVINO support):
# /dev/dri/* device nodes must exist
ls /dev/dri
# expected: card0 renderD128
# host user needs video + render group membership for /dev/dri access
sudo usermod -aG video,render "$USER"
# log out / log back in so it takes effect
# confirm
groups | grep -oE 'video|render'
# (optional) verify Intel compute runtime is installed on the host - the
# container ships its own copy, so this is just a sanity check
clinfo 2>/dev/null | grep -i 'intel.*gpu' || echo "(host clinfo not installed; container will provide its own)"If you have no GPU, set ILLAMA_DEVICE=CPU (much slower; works for smoke tests).
# 1. Clone the repo onto the host
git clone https://github.com/mkhomutskyi/illama2.git ~/illama2
cd ~/illama2
# 2. Run the installer. This:
# - creates /opt/illama/{models,cache,config,deploy,...}
# - copies the runner sources, server config, deploy artefacts
# - builds illama-runner:latest + illama-runner:<version>
# - drops a symlink at /usr/local/bin/illama
chmod +x setup.sh
./setup.sh
# 3. Pull a pre-converted OpenVINO model (~2-5 min depending on size)
illama pull OpenVINO/Phi-4-int4-ov
# 4. Pick ONE of the entry points below.illama serve # starts illama-service container
illama ps # status + loaded model
illama run Phi-4-int4-ov -p "Hi" # smoke-test inference
illama logs # tail server logs
illama stop # stop containersetup.sh copies a ready-to-go compose file and an .env (seeded from .env.example on first install) to /opt/illama/deploy/.
cd /opt/illama/deploy
# 1. (optional) review/tune the env. Defaults match `illama serve`.
$EDITOR .env
# 2. Bring it up
docker compose up -d
# 3. Verify
docker compose ps
docker compose logs -f illama-runner # Ctrl-C to detach
curl -s http://localhost:11434/illama/health | python3 -m json.tool
# 4. Stop / restart on config change
docker compose down
docker compose up -dThe compose file mirrors illama serve exactly: same image, same volumes (/opt/illama/{models:ro,config:ro,cache}), same --device /dev/dri, same --group-add for video/render, same healthcheck. See deploy/docker-compose.yml for the canonical reference.
Layered on top of compose; the unit just runs docker compose up -d on boot and down on stop.
sudo cp /opt/illama/deploy/illama-runner.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now illama-runner
systemctl status illama-runner
journalctl -u illama-runner -f # live logsSwitch back to manual control:
sudo systemctl disable --now illama-runner
illama serveEvery knob is defined in deploy/.env.example and consumed by deploy/docker-compose.yml. Copy the example to deploy/.env (setup.sh does this for you on first install) and edit:
| Variable | Default | Purpose |
|---|---|---|
ILLAMA_RUNNER_TAG |
latest |
Image tag to launch. Pin to a version (2.3.0) for immutable deploys. |
ILLAMA_HOME |
/opt/illama |
Base dir holding models, config, cache. |
ILLAMA_PORT |
11434 |
Host-side port; matches Ollama's default for OpenWebUI compatibility. |
ILLAMA_DEVICE |
GPU |
GPU (Intel via OpenVINO) or CPU. |
HOST_VIDEO_GID |
44 |
GID of host video group; check with getent group video. |
HOST_RENDER_GID |
110 |
GID of host render group; check with getent group render. |
ILLAMA_IDLE_TIMEOUT |
(falls back to config) |
When to unload model from VRAM. 300, 300s, 5m, or 0 to disable. |
ILLAMA_API_KEY |
empty | Optional bearer-token auth. Generate with openssl rand -hex 32. |
ILLAMA_ALLOW_PRIVATE_IMAGE_URLS |
empty | 1 to disable VLM-image SSRF guard (only on trusted private networks). |
ILLAMA_VERSION |
(baked at build) |
Override the version surfaced by /api/version. |
Container-internal settings live in /opt/illama/config/server.json (max_new_tokens_chat, max_input_tokens, allow_stream, etc.) — see Configuration. The compose file mounts config/ read-only, so edit on the host and docker compose restart illama-runner to pick up changes.
After any of the three entry points are up:
# 1. Health (process + model status)
curl -s http://localhost:11434/illama/health | python3 -m json.tool
# 2. Confirm CACHE_DIR is wired through (saves ~10s+ on subsequent loads)
docker logs illama-service 2>&1 | grep -E 'Cache dir|Pipeline properties'
# Expected: "[INFO] Cache dir: /cache"
# "[INFO] Pipeline properties: {'CACHE_DIR': '/cache'}"
# 3. OpenAI-compatible non-stream + stream chat
illama run Phi-4-int4-ov -p "Reply with: hello"
illama run Phi-4-int4-ov -p "Count from 1 to 5" --stream
# 4. Prometheus metrics
curl -s http://localhost:11434/metrics | head -20
# 5. (recommended) host-level diagnostics
illama doctorIf anything is off, illama doctor checks Docker, GPU device nodes, group membership, registry contents, and reachability of the runner — and prints concrete fixes.
setup.sh is idempotent. Models in /opt/illama/models and the GPU compile cache in /opt/illama/cache are preserved. Live config/server.json is preserved too — if the shipped defaults change, the new file lands at config/server.json.new so you can diff and merge.
setup.sh copies this repo's Dockerfiles over /opt/illama and rebuilds the runner from those files. It does not bump libraries past the pins in the Dockerfiles, and it does not rebuild the builder image. To move OpenVINO or optimum-intel to a newer release, follow How to update runtime libraries.
# 1. Sync new sources
cd ~/illama2
git pull
# 2. Re-run the installer. Rebuilds runner image with the new VERSION tag.
./setup.sh
# 3. Restart whichever entry point you use
illama stop && illama serve
# or
cd /opt/illama/deploy && docker compose up -d --force-recreate
# or
sudo systemctl restart illama-runner
# 4. Verify
illama ps
illama doctorsetup.sh tags every built image as both illama-runner:latest and illama-runner:<version> so a rollback is one tag move:
# Pin :latest back to the previous version
docker tag illama-runner:2.1.0 illama-runner:latest
illama stop && illama serve
# Or via compose, point .env at the older tag
echo 'ILLAMA_RUNNER_TAG=2.1.0' >> /opt/illama/deploy/.env
cd /opt/illama/deploy && docker compose up -dTo revert to an older source tree as well:
cd ~/illama2
git checkout <previous-tag>
./setup.shillama stop # or: docker compose down
sudo systemctl disable --now illama-runner 2>/dev/null || true
sudo rm -f /etc/systemd/system/illama-runner.service /usr/local/bin/illama
docker rmi illama-runner:latest illama-builder:latest 2>/dev/null || true
sudo rm -rf /opt/illama # also wipes models + cache; back up first if needed- Architecture
- Components
- Directory Structure
- Suggested Models
- Quick Start
- Commands
- Cache Management
- Validated Models
- Configuration
- Upgrade & Backup
- Remote Server Update
- OpenWebUI Integration
- Requirements
- Troubleshooting
- OpenVINO Dependencies
- How to update runtime libraries
- Third-party stacks
- License
- B50 notes: docs/b50-model-notes.md — GPU passthrough gotchas, benchmarks, EAGLE-3
Illama uses a decoupled architecture with two main components:
┌─────────────────────────────────────┐
│ Ubuntu Host │
│ │
┌──────────────┐ │ ┌─────────────────────────────┐ │
│ User CLI │ │ │ illama-runner │ │
│ │ illama serve │ │ (FastAPI + GenAI) │ │
│ $ illama │───────────────────▶│ │ │ │
│ │ │ │ ┌─────────────────────┐ │ │
└──────────────┘ │ │ │ OpenVINO GenAI │ │ │
│ │ │ │ (LLMPipeline) │ │ │
│ │ │ └─────────────────────┘ │ │
│ illama pull │ │ │ │ │
│ │ │ ▼ │ │
▼ │ │ ┌─────────────────────┐ │ │
┌──────────────┐ │ │ │ Intel GPU │ │ │
│illama-builder│ │ │ │ /dev/dri │ │ │
│ (Ephemeral) │ │ │ └─────────────────────┘ │ │
│ │ │ └─────────────────────────────┘ │
│ optimum-cli │ │ ▲ │
│ export │ │ │ │
│ openvino │──────────────────▶ │ ┌────────────┴────────────┐ │
└──────────────┘ Writes models │ │ /opt/illama/models │ │
│ │ (Shared Volume) │ │
│ └─────────────────────────┘ │
└────────────────────────────────────┘
- Model Pull:
illama pull→ spawns ephemeralillama-builder→ downloads from HuggingFace → converts to OpenVINO IR (INT4) → writes to/opt/illama/models/ - Inference:
illama serve→ startsillama-runnercontainer → loads models on-demand → exposes OpenAI-compatible API - Idle Management: Built into server → unloads model from memory after configurable idle timeout
Purpose: Converts HuggingFace models to OpenVINO IR format with INT4 quantization.
| Property | Value |
|---|---|
| Base Image | ubuntu:24.04 |
| Key Packages | optimum-intel, transformers, nncf, accelerate |
| Lifecycle | Starts → Converts → Exits (not persistent) |
| Entrypoint | optimum-cli export openvino |
Why isolated? The conversion dependencies (torch, transformers, nncf) are heavy and conflict-prone. By isolating them, breaking changes in HuggingFace never crash your running server.
Purpose: Lightweight FastAPI server using OpenVINO GenAI for LLM inference.
| Property | Value |
|---|---|
| Image | illama-runner:latest (built locally) |
| Base | ubuntu:24.04 + openvino-genai + fastapi |
| API Port | Configurable (default: 11434) |
| OpenAI surface | /v1/models, /v1/chat/completions, /v1/completions, /v1/unload |
| Ollama surface | /api/version, /api/tags, /api/ps, /api/show, /api/chat, /api/generate |
| Health / admin | /health, /illama/health, /illama/stats, /metrics |
| Streaming | SSE for /v1/*, NDJSON for /api/* |
| Lifecycle | Persistent service (restarts on reboot) |
Features:
- Single model loaded at a time (memory efficient)
- Built-in idle unload (no external watchdog needed)
- GPU acceleration via Intel OpenCL/Level Zero, with persistent OpenVINO GPU model cache at
/cache - OpenAI-compatible API with SSE streaming, plus a parallel Ollama-compatible NDJSON surface for clients like Continue.dev, LangChain
ChatOllama, and OpenWebUI's Ollama backend - Optional bearer-token auth (timing-safe comparison)
- Prometheus-style
/metricsendpoint for lightweight monitoring - Stats persisted to disk so counters survive restarts
Purpose: Orchestrates builder and runner containers, manages the model registry, and provides ergonomic local commands. See the Commands section below for the full table; in summary the CLI handles serve/stop/ps/logs, pull/list/rm/show, run (interactive or single-shot, with optional streaming), bench, prune, and version.
After installation, the following directory structure is created:
/opt/illama/ # ILLAMA_HOME (configurable)
├── builder/
│ └── Dockerfile # Builder image definition
├── runner/
│ ├── Dockerfile # Runner image definition
│ └── server.py # FastAPI server code
├── config/
│ └── server.json # Runner configuration
├── models/ # MODEL REGISTRY
│ ├── .cache/ # HuggingFace cache (during pull)
│ ├── Qwen2.5-7B-Instruct/ # Example model
│ │ ├── config.json # Model configuration
│ │ ├── generation_config.json # Generation parameters
│ │ ├── openvino_model.bin # Model weights (large file)
│ │ ├── openvino_model.xml # Model graph definition
│ │ ├── openvino_tokenizer.bin # Tokenizer weights
│ │ ├── openvino_tokenizer.xml # Tokenizer graph
│ │ ├── openvino_detokenizer.bin # Detokenizer weights
│ │ ├── openvino_detokenizer.xml # Detokenizer graph
│ │ ├── tokenizer.json # Original tokenizer
│ │ ├── tokenizer_config.json # Tokenizer settings
│ │ └── special_tokens_map.json # Special tokens
│ └── TinyLlama-1.1B-Chat-v1.0/ # Another model
│ └── ...
├── cache/ # OpenVINO GPU model compile cache + stats snapshot
│ └── illama_stats.json # Persisted /illama/stats counters (survive restart)
└── logs/ # Log files (optional)
Each model folder in /opt/illama/models/ contains:
| File | Purpose | Size (typical) |
|---|---|---|
openvino_model.bin |
Quantized model weights | 3-8 GB |
openvino_model.xml |
Model graph (OpenVINO IR) | ~1 MB |
openvino_tokenizer.* |
Tokenizer for text → tokens | ~2 MB |
openvino_detokenizer.* |
Detokenizer for tokens → text | ~2 MB |
config.json |
Model architecture config | ~2 KB |
tokenizer.json |
Full tokenizer vocabulary | ~2 MB |
model.json |
Illama metadata and optional per-model overrides | small |
Models tested and recommended for use with Illama on Intel GPUs:
| VRAM | Model | HuggingFace Repo | INT4 Size |
|---|---|---|---|
| 8 GB | TinyLlama 1.1B | TinyLlama/TinyLlama-1.1B-Chat-v1.0 |
~600 MB |
| 8 GB | Llama 3.2 3B | meta-llama/Llama-3.2-3B-Instruct |
~2 GB |
| 8 GB | Qwen2.5 3B | Qwen/Qwen2.5-3B-Instruct |
~2 GB |
| 12 GB | Qwen2.5 7B | Qwen/Qwen2.5-7B-Instruct |
~4 GB |
| 12 GB | Gemma 3 12B | google/gemma-3-12b-it |
~6 GB |
| 12 GB | MiniCPM-o 2.6 | openbmb/MiniCPM-o-2_6 |
~6 GB |
| 16 GB | Llama 3.2 | meta-llama/Llama-3.2-11B-Vision-Instruct |
~6 GB |
| 16 GB | Phi-4 14B | microsoft/Phi-4 |
~8 GB |
| 16 GB | OLMo 3 7B | allenai/OLMo-2-1124-7B-Instruct |
~8 GB |
| 16 GB | GPT-OSS 20B | openai/gpt-oss-20b |
~10 GB |
| 24 GB | Qwen3 30B (MoE) | Qwen/Qwen3-30B-A3B-Instruct |
~10 GB |
| 24 GB | Qwen3 Coder 30B | Qwen/Qwen3-Coder-30B-A3B-Instruct |
~10 GB |
| Use Case | Recommended Model | HuggingFace Repo |
|---|---|---|
| General Chat | Qwen2.5 7B | Qwen/Qwen2.5-7B-Instruct |
| Coding | Qwen3 Coder 30B | Qwen/Qwen3-Coder-30B-A3B-Instruct |
| Fast Testing | TinyLlama 1.1B | TinyLlama/TinyLlama-1.1B-Chat-v1.0 |
| Reasoning | Phi-4 14B | microsoft/Phi-4 |
| Multilingual | Qwen3 30B | Qwen/Qwen3-30B-A3B-Instruct |
| Vision/Audio | MiniCPM-o 2.6 | openbmb/MiniCPM-o-2_6 |
| Vision | Llama 3.2 Vision | meta-llama/Llama-3.2-11B-Vision-Instruct |
| OpenAI Compatible | GPT-OSS 20B | openai/gpt-oss-20b |
| Fully Open Source | OLMo 3 | allenai/OLMo-2-1124-7B-Instruct |
illama pull accepts either the canonical HuggingFace repo id (org/name)
or a full HuggingFace URL — paste-from-browser works the same as paste-from-search:
# All four of these are equivalent:
illama pull OpenVINO/Phi-4-int4-ov
illama pull https://huggingface.co/OpenVINO/Phi-4-int4-ov
illama pull https://huggingface.co/OpenVINO/Phi-4-int4-ov/tree/main
illama pull hf://OpenVINO/Phi-4-int4-ovA trailing /tree/<ref>, /blob/<ref>/<path>, /resolve/<ref>/<path>,
query string, or #fragment is stripped automatically; the runner only
ever sees the canonical org/name.
# Small & fast (for testing)
illama pull TinyLlama/TinyLlama-1.1B-Chat-v1.0
# Llama 3.2 (small)
illama pull meta-llama/Llama-3.2-3B-Instruct
# Best all-around (7B)
illama pull Qwen/Qwen2.5-7B-Instruct
# Google Gemma 3
illama pull google/gemma-3-12b-it
# Microsoft Phi-4 (great reasoning)
illama pull microsoft/Phi-4
# OpenAI GPT-OSS (working in OpenVINO 2026)
illama pull openai/gpt-oss-20b
# AllenAI OLMo 3 (fully open source)
illama pull allenai/OLMo-2-1124-7B-Instruct
# Qwen3 MoE (best quality, needs 24GB)
illama pull Qwen/Qwen3-30B-A3B-Instruct
# Qwen3 Coder (for coding)
illama pull Qwen/Qwen3-Coder-30B-A3B-InstructSome models (Llama, Gemma, etc.) are "gated" and require:
- Accepting the license on HuggingFace
- Providing your HF token
To authenticate:
# Get your token from https://huggingface.co/settings/tokens
# Option 1: Set for current session
export HF_TOKEN="hf_your_token_here"
illama pull meta-llama/Llama-3.2-3B-Instruct
# Option 2: Add to ~/.bashrc for permanent use
echo 'export HF_TOKEN="hf_your_token_here"' >> ~/.bashrc
source ~/.bashrcBefore running setup.sh, you can configure the default port and idle timeout:
# Set custom port (default is 11434)
export ILLAMA_PORT=11435
# Override idle timeout if you don't want the default 60s
export ILLAMA_IDLE_TIMEOUT=600 # 10 minutes
# Make permanent (add to ~/.bashrc)
echo 'export ILLAMA_PORT=11435' >> ~/.bashrc
echo 'export ILLAMA_IDLE_TIMEOUT=600' >> ~/.bashrc
source ~/.bashrcchmod +x setup.sh
./setup.shillama pull Qwen/Qwen2.5-7B-Instructillama serveillama run Qwen2.5-7B-Instruct| Command | Description | Example |
|---|---|---|
illama serve |
Start the inference server | illama serve |
illama stop |
Stop the server | illama stop |
illama ps |
Show server status & models | illama ps |
illama pull <repo> |
Download & convert a model | illama pull Qwen/Qwen2.5-7B-Instruct |
illama search [query] [--vlm] |
Browse the HuggingFace OpenVINO model index | illama search gemma --vlm |
illama list |
List installed models | illama list |
illama rm <model> |
Delete a model | illama rm Qwen2.5-7B-Instruct |
illama show <model> |
Show model info | illama show Qwen2.5-7B-Instruct |
illama run <model> |
Interactive chat | illama run Qwen2.5-7B-Instruct |
illama run <model> --stream |
Stream output live in the terminal | illama run phi-4-int4-ov --stream |
illama logs [N] |
Tail runner logs (default 100, follows) | illama logs 200 |
illama bench [args...] |
Benchmark loaded/all models via the API | illama bench --models Qwen2.5-7B-Instruct |
illama prune builder |
Remove the conversion image (~5 GB) | illama prune builder |
illama gc [--dry-run] [--force] |
Reclaim disk: prune HF cache + broken model dirs | illama gc --dry-run |
illama doctor |
Run host / runtime health checks (no changes) | illama doctor |
illama version |
Show version info | illama version |
illama --help |
Show help | illama --help |
illama -v <cmd> |
Verbose mode | illama -v serve |
In streamed CLI mode, Illama now requests stream usage data and prints the same token/speed summary at the end of the streamed reply.
illama search queries the public HuggingFace API (/api/models?filter=openvino), prints a numbered table of trending OpenVINO repos, and offers to one-step pull whichever entry you pick.
illama search # 20 trending OpenVINO repos
illama search gemma # filtered by 'gemma'
illama search gemma --vlm # only Vision Language Models (image-text-to-text)
illama search --sort downloads --limit 30Output looks like:
# NAME TYPE DLs LIKES UPDATED
1 OpenVINO/gemma-4-E4B-it-int4-ov image-text-to-text 976 2 13 days ago
2 OpenVINO/gemma-4-E4B-it-int8-ov image-text-to-text 339 2 13 days ago
...
Pull which one? [1-N / Enter to skip]:
Pre-converted OpenVINO/... repos are pulled via huggingface_hub.snapshot_download directly. Other repos are converted with the builder image (so they go through optimum-cli export openvino --weight-format int4).
If you set HF_TOKEN in your environment, the search request will pass it as a Bearer token so gated repos show up too.
During model download and conversion, HuggingFace caches files in /opt/illama/models/.cache/. This can consume significant disk space.
# Total cache size
du -sh /opt/illama/models/.cache/
# Cache breakdown by model
du -sh /opt/illama/models/.cache/huggingface/hub/*/The cache is automatically cleaned after successful conversions. To manually clean:
# Clean all cache (safe after conversions are complete)
sudo rm -rf /opt/illama/models/.cache/*
# Check disk space freed
df -h /opt/illama/models/Find models that failed conversion (missing OpenVINO files):
for d in /opt/illama/models/*/; do
name=$(basename "$d")
[[ "$name" == ".cache" ]] && continue
if [[ ! -f "$d/openvino_model.xml" ]] && [[ ! -f "$d/openvino_language_model.xml" ]]; then
echo "Incomplete: $name ($(du -sh "$d" | cut -f1))"
fi
done# List all models with disk usage
illama list
# Detailed view
du -sh /opt/illama/models/*/ | sort -hIf containers are missing or corrupted:
cd /opt/illama
# The version comes from the VERSION file at the repo root and is also
# tagged by setup.sh so you can roll back without rebuilding from a
# different git ref:
# docker tag illama-runner:<previous-version> illama-runner:latest
# # e.g. after upgrading to 2.3.0:
# # docker tag illama-runner:2.2.0 illama-runner:latest
ILLAMA_VERSION="$(tr -d '[:space:]' < /opt/illama/VERSION 2>/dev/null || echo dev)"
# Rebuild builder (tagged :latest and :$ILLAMA_VERSION)
docker build \
--build-arg "ILLAMA_VERSION=${ILLAMA_VERSION}" \
-t illama-builder:latest \
-t "illama-builder:${ILLAMA_VERSION}" \
builder/
# Rebuild runner (tagged :latest and :$ILLAMA_VERSION)
docker build \
--build-arg "ILLAMA_VERSION=${ILLAMA_VERSION}" \
-t illama-runner:latest \
-t "illama-runner:${ILLAMA_VERSION}" \
runner/
# Verify both exist
docker images | grep illamaThe tests/ directory contains a fast pure-Python smoke suite for the
helpers in runner/server.py (SSRF guard, base64 decoding, multimodal
content flattening, config parsing, version resolution, OpenAI error
shape, etc.). It runs on a vanilla Python install with pytest and
fastapi + pydantic only — openvino, PIL, numpy, and
requests are stubbed in tests/conftest.py so CI does not need the
Intel runtime.
pip install pytest fastapi pydantic
pytest tests/ -vUse the included benchmark script against the running OpenAI-compatible API:
python3 scripts/benchmark_api.py \
--base-url http://localhost:${ILLAMA_PORT:-11434}/v1 \
--models phi-4-int4-ov Qwen3-14B-int4-ov gpt-oss-20b \
--cold-start \
--csv benchmarks/latest.csv \
--markdown benchmarks/latest.mdIf auth is enabled:
python3 scripts/benchmark_api.py \
--base-url http://localhost:${ILLAMA_PORT:-11434}/v1 \
--api-key "${ILLAMA_API_KEY}" \
--cold-startThe script measures:
- short non-stream latency
- long non-stream latency and throughput
- stream time-to-first-token
- stream total completion time
Models tested and confirmed working with Illama on Intel Arc GPU (15GB VRAM):
| Model | Size | Device | Notes |
|---|---|---|---|
TinyLlama-1.1B-Chat-v1.0 |
632M | GPU | ⚡ Fast, good for testing |
gemma-3-4b-it-int4-ov |
3.3G | GPU | OpenVINO pre-converted |
Llama-3.1-8B-Instruct |
5.2G | GPU | Reliable |
gemma-3-12b-it-qat-q4_0-unquantized |
7.6G | GPU | |
phi-4-int4-ov |
7.6G | GPU | OpenVINO pre-converted |
DeepSeek-R1-Distill-Qwen-14B-int4-ov |
7.9G | GPU | ✨ Recommended |
Qwen3-14B-int4-ov |
9.1G | GPU | OpenVINO pre-converted |
gpt-oss-20b |
12G | GPU | ✨ Working in OpenVINO 2026 |
gemma-3-27b-it |
16G | GPU | |
Qwen3-30B-A3B-Instruct-2507 |
16G | CPU | MoE - runs on CPU |
Qwen3-Coder-30B-A3B-Instruct |
16G | CPU | MoE - runs on CPU |
GPU Memory Guide: Models up to 14B are the comfortable tier on 15-16GB Intel Arc/Battlemage. Some larger models such as
gemma-3-27b-itcan still run on GPU, but they leave far less headroom and feel much slower without streaming.
Large-Model UX Note: Illama now supports streaming, which helps a lot for large models. If a client disables streaming or buffers the response, models like
gemma-3-27b-itcan still take tens of seconds before the full reply is visible.
The following practical tiers are based on live benchmarking on an Intel Battlemage B50 using Illama's OpenAI-compatible API.
Recommended defaults
Qwen3-8B-int4-ov: best balanced default for interactive chatgpt-oss-20b-int4-ov: highest upside from the tested set; surprisingly strong throughput on B50DeepSeek-R1-Distill-Qwen-14B-int4-ov: strong reasoning-focused defaultQwen3-14B-int4-ov: strong all-around 14B option
Good but more situational
phi-4-int4-ov: solid, but not clearly ahead of the best 14B options in these testsgemma-3-12b-it-qat-q4_0-unquantized: usable, but not as compelling as DeepSeek/Qwen3-14BQwen3-30B-A3B-Instruct-2507: good throughput after startup, but slower first tokenQwen3-Coder-30B-A3B-Instruct: strong coding candidate if slower startup is acceptable
Experimental / outlier tier
gemma-3-27b-it: works, but should be treated as an outlier model with smaller per-model token budgetsTinyLlama-1.1B-Chat-v1.0: useful smoke-test model, not a primary production choice
Start with these overrides for the slow outliers:
For gemma-3-27b-it:
{
"overrides": {
"max_new_tokens_chat": 512,
"max_new_tokens_completion": 512,
"recommended_stream": true
}
}For Qwen3-30B-A3B-Instruct-2507:
{
"overrides": {
"max_new_tokens_chat": 1024,
"max_new_tokens_completion": 1024,
"recommended_stream": true
}
}For Qwen3-Coder-30B-A3B-Instruct:
{
"overrides": {
"max_new_tokens_chat": 1024,
"max_new_tokens_completion": 1024,
"recommended_stream": true
}
}The global defaults can remain at 2048; only a small number of large models need tighter per-model limits.
Runtime defaults live in config/server.json. Environment variables override the most common deployment settings:
| Variable | Default | Description |
|---|---|---|
ILLAMA_HOME |
/opt/illama |
Base directory for all illama files |
ILLAMA_PORT |
11434 |
API server port (OpenAI-compatible) |
ILLAMA_IDLE_TIMEOUT |
60 |
Inactivity before auto-unload. Accepts seconds (60), seconds with unit (60s), or minutes (1m). 0 = disabled. |
ILLAMA_API_KEY |
unset | Optional bearer token required by /v1/*, /illama/*, and /metrics when set. |
ILLAMA_LOAD_FAILURE_POLICY |
unset | Optional override for load_failure_policy (restart or return_error). |
Server-side request defaults and limits live in config/server.json.
Example:
{
"idle_timeout": "60s",
"allow_stream": true,
"max_new_tokens_chat": 2048,
"max_new_tokens_completion": 2048,
"max_messages_per_request": 128,
"max_prompt_chars": 131072,
"max_input_tokens": 0,
"max_stop_sequences": 8,
"max_stop_sequence_length": 256,
"load_failure_policy": "restart"
}Knobs:
| Key | Default | Description |
|---|---|---|
idle_timeout |
60s |
Unload the currently loaded model after this idle period. |
allow_stream |
true |
Enables stream=true SSE responses. Set to false only if you want to force non-streaming behavior. |
max_new_tokens_chat |
2048 |
Default max_tokens for /v1/chat/completions when the client does not supply one. |
max_new_tokens_completion |
2048 |
Default max_tokens for /v1/completions when the client does not supply one. |
max_messages_per_request |
128 |
Upper bound for the number of chat messages accepted in one request. |
max_prompt_chars |
131072 |
Maximum allowed prompt size in characters before tokenization. |
max_input_tokens |
0 |
Maximum allowed input tokens after tokenization. 0 disables the token limit. |
max_stop_sequences |
8 |
Maximum number of stop sequences accepted in a request. |
max_stop_sequence_length |
256 |
Maximum character length of any single stop sequence. |
load_failure_policy |
restart |
restart exits the process so Docker can restart cleanly. return_error keeps the server alive and returns an API error when a model load fails. |
Set ILLAMA_API_KEY only if you want bearer-token auth enabled.
export ILLAMA_API_KEY="sk-local-secret"
illama serveWhen auth is enabled:
- OpenWebUI should use the same value in its
API Keyfield - raw clients must send
Authorization: Bearer <key> /,/health, and docs remain open for basic liveness checks
Example:
curl http://localhost:${ILLAMA_PORT:-11434}/v1/models \
-H "Authorization: Bearer ${ILLAMA_API_KEY}"Global defaults stay in config/server.json, but a model can override a small approved subset of request-budget knobs through models/<model>/model.json.
Resolution order is:
- explicit request value
- per-model override in
model.json - global value from
config/server.json - hardcoded fallback
Supported per-model override keys:
max_new_tokens_chatmax_new_tokens_completionmax_input_tokensmax_prompt_charsrecommended_stream
Example for a large outlier model:
{
"repo_id": "google/gemma-3-27b-it",
"source": "converted-openvino",
"overrides": {
"max_new_tokens_chat": 512,
"max_new_tokens_completion": 512
}
}This is intended for a small number of exceptional models such as gemma-3-27b-it or gpt-oss-20b, while the global defaults remain at 2048.
The effective request limits for the currently loaded model are exposed through:
/illama/health/illama/stats
# Temporary (current session)
export ILLAMA_PORT=8080
illama serve
# Permanent (add to ~/.bashrc or ~/.profile)
echo 'export ILLAMA_PORT=8080' >> ~/.bashrc
source ~/.bashrc
illama serveAfter changing port, update OpenWebUI connection:
API Base URL: http://<your-ip>:8080/v1
The idle timeout controls how long a model can sit unused in memory before it is automatically unloaded (the server process keeps running).
# Set to 10 minutes (600 seconds)
export ILLAMA_IDLE_TIMEOUT=600
illama serve
# Disable idle unload (server keeps model loaded)
export ILLAMA_IDLE_TIMEOUT=0
illama serve
# Set to 1 minute
export ILLAMA_IDLE_TIMEOUT=60
illama serveYou can also use a more human-friendly format:
export ILLAMA_IDLE_TIMEOUT=5m # 5 minutes
export ILLAMA_IDLE_TIMEOUT=30s # 30 secondsHow it works:
- When
illama serveruns, the FastAPI server starts an internal idle watchdog thread. - The watchdog samples
last_request_timeroughly four times perILLAMA_IDLE_TIMEOUTwindow (capped between 1s and 30s), so a 60s timeout fires within ~75s wallclock and a 15s timeout fires within ~18s wallclock. - If there are no new requests for
ILLAMA_IDLE_TIMEOUTseconds, it unloads the currently loaded model from memory (VRAM/CPU RAM). - The container stays running; the next
illama runor OpenAI API call automatically reloads the model on demand.
Inspecting the unload schedule: GET /api/ps returns each currently-resident model and its expires_at timestamp (= last_request_time + idle_timeout). When idle_timeout is 0 the field is set to 9999-12-31T23:59:59.000Z, matching what stock Ollama emits for keep_alive=-1 so existing dashboard clients render it as "Forever".
Illama now returns OpenAI-style error objects for request validation and runtime failures:
{
"error": {
"message": "Input exceeds maximum length of 131072 characters",
"type": "invalid_request_error",
"param": "messages",
"code": "input_too_large"
}
}Typical cases include:
- invalid request bodies
- input size or token-limit violations
- model lookup failures
- generation errors
# Install to custom location
export ILLAMA_HOME=/data/illama
./setup.sh
# All subsequent commands use this path
export ILLAMA_HOME=/data/illama
illama serveCreate /etc/profile.d/illama.sh:
#!/bin/bash
export ILLAMA_HOME=/opt/illama
export ILLAMA_PORT=11434
export ILLAMA_IDLE_TIMEOUT=1800 # 30 minutesThen:
sudo chmod +x /etc/profile.d/illama.sh
source /etc/profile.d/illama.shThis section shows how to safely upgrade Illama to a new version without losing your downloaded models.
On your server:
# Stop the running service (optional but recommended)
illama stop || true
# Create a backup directory in your home
mkdir -p ~/illama-backups
# Archive the entire /opt/illama tree (includes models)
sudo tar czf \
~/illama-backups/illama-$(date +%Y%m%d-%H%M).tgz \
/opt/illamaThis tarball contains:
- All converted models in
/opt/illama/models - Dockerfiles,
server.py, CLI script, config files, logs, etc.
Optional: also back up current Docker images:
docker save illama-runner:latest \
-o ~/illama-backups/illama-runner-$(date +%Y%m%d-%H%M).tar
docker save illama-builder:latest \
-o ~/illama-backups/illama-builder-$(date +%Y%m%d-%H%M).tarOn the same server, update your Illama source repo (e.g. ~/illama2):
cd ~/illama2
git pull # or copy the new files into this directoryThen re-run the setup script:
cd ~/illama2
chmod +x setup.sh
./setup.shWhat setup.sh does:
- Copies updated
builder/Dockerfile,runner/Dockerfile,runner/server.py, andillamainto/opt/illama. - Rebuilds
illama-builder:latestandillama-runner:latest. - Refreshes the
/usr/local/bin/illamasymlink. - Does not delete
/opt/illama/models, so all previously pulled models remain on disk.
illama serve
# Check status
illama ps
# Verify API is up
curl http://localhost:${ILLAMA_PORT:-11434}/v1/models | jq
# Optionally check stats
curl http://localhost:${ILLAMA_PORT:-11434}/illama/health | jq
curl http://localhost:${ILLAMA_PORT:-11434}/illama/stats | jqYour existing models should appear in both illama list and /v1/models without needing to re-download.
If something goes wrong and you want to roll back:
illama stop || true
# Restore /opt/illama from a previous backup
sudo rm -rf /opt/illama
sudo tar xzf ~/illama-backups/illama-YYYYMMDD-HHMM.tgz -C /
# Optionally restore old Docker images
docker load -i ~/illama-backups/illama-runner-YYYYMMDD-HHMM.tar
docker load -i ~/illama-backups/illama-builder-YYYYMMDD-HHMM.tar
# Start the restored version
illama serveThis guarantees that both your code and all downloaded models are restored to a known-good state.
This section is the practical step-by-step update path for a remote Illama server.
Run these on the remote server before changing anything:
echo "illama binary: $(command -v illama || echo not-found)"
readlink -f "$(command -v illama 2>/dev/null)" 2>/dev/null || true
echo "ILLAMA_HOME=${ILLAMA_HOME:-<unset>}"
echo "ILLAMA_PORT=${ILLAMA_PORT:-<unset>}"
illama version || true
docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'
docker images | grep -E 'illama|openvino' || true
ls -ld /opt/illama /data/illama 2>/dev/null || trueIf ILLAMA_HOME is unset, the default expected install location is /opt/illama.
illama stop || true
mkdir -p ~/illama-backups
sudo tar czf ~/illama-backups/illama-$(date +%Y%m%d-%H%M).tgz /opt/illama
docker save illama-runner:latest -o ~/illama-backups/illama-runner-$(date +%Y%m%d-%H%M).tar || true
docker save illama-builder:latest -o ~/illama-backups/illama-builder-$(date +%Y%m%d-%H%M).tar || trueIf your live install is not in /opt/illama, replace that path with the discovered ILLAMA_HOME.
On the remote server, place the new source in a working directory such as ~/illama2, then:
cd ~/illama2
git pull || trueIf you are not using git on the server, copy the new project files into that working directory instead.
sudo cat /opt/illama/config/server.json 2>/dev/null || trueCheck whether the live server has custom values for:
idle_timeoutallow_streammax_new_tokens_chatmax_new_tokens_completion- request limit knobs such as
max_prompt_charsormax_input_tokens
If needed, save a copy first:
sudo cp /opt/illama/config/server.json ~/illama-backups/server.json.$(date +%Y%m%d-%H%M) 2>/dev/null || truecd ~/illama2
chmod +x setup.sh
./setup.shWhat this does:
- copies the current
builder/,runner/,config/server.json, andillamainto the live install directory, overwriting/opt/illama/runner/Dockerfileand/opt/illama/builder/Dockerfile - rebuilds
illama-runner:latest(the builder image is not rebuilt here;illama pullbuilds it on demand, or build it explicitly as in How to update runtime libraries) - refreshes
/usr/local/bin/illama - keeps the existing model files under
ILLAMA_HOME/models
Run this only when the git Dockerfiles already contain the library pins you want. A sed edit made directly on /opt/illama is replaced by whatever this checkout pins.
illama version
docker run --rm illama-runner:latest python3 -c "import openvino, openvino_genai; print(openvino.__version__, openvino_genai.__version__)"
docker run --rm illama-builder:latest python3 -c "import transformers, optimum; print(transformers.__version__)"
illama serve
illama ps
curl http://localhost:${ILLAMA_PORT:-11434}/illama/health
curl http://localhost:${ILLAMA_PORT:-11434}/v1/modelsillama run TinyLlama-1.1B-Chat-v1.0 -p "hello"If that model is not installed, use any small model already present on the server:
illama listIn OpenWebUI, confirm the connection still points to:
http://<server-ip>:11434/v1
Then refresh models and send one test prompt.
illama stop || true
sudo rm -rf /opt/illama
sudo tar xzf ~/illama-backups/illama-YYYYMMDD-HHMM.tgz -C /
docker load -i ~/illama-backups/illama-runner-YYYYMMDD-HHMM.tar || true
docker load -i ~/illama-backups/illama-builder-YYYYMMDD-HHMM.tar || true
illama serveIllama exposes two parallel API surfaces against the same inference engine:
| Path prefix | Schema | Streaming | Best for |
|---|---|---|---|
/v1/* |
OpenAI Chat Completions | SSE (text/event-stream) |
OpenAI clients, OpenWebUI in OpenAI mode, LangChain ChatOpenAI, etc. |
/api/* |
Ollama | NDJSON (application/x-ndjson) |
Continue.dev, LangChain ChatOllama, OpenWebUI in Ollama mode, anything that hard-codes Ollama's URL shape |
curl -s http://localhost:11434/api/version
# {"version":"2.3.0"}
curl -s http://localhost:11434/api/tags | python3 -m json.tool
# Streaming chat
curl -sN http://localhost:11434/api/chat -d '{
"model": "Qwen2.5-7B-Instruct",
"messages": [{"role":"user","content":"Hello"}],
"stream": true
}'The Ollama subset implemented: GET /api/version, GET /api/tags, GET /api/ps,
POST /api/show, POST /api/chat, POST /api/generate. Operations that don't
make sense in illama's "models live on disk, conversion is explicit" model
(POST /api/pull, POST /api/create, DELETE /api/delete) are not
implemented — use illama pull / illama rm for those.
seedis honored (mapped toGenerationConfig.rng_seed).response_format={"type":"json_object"}prepends a system instruction asking for JSON output (best-effort — there is no constrained sampling in OpenVINO GenAI yet).frequency_penalty > 0is mapped torepetition_penalty = 1 + frequency_penalty.presence_penalty,n,logprobsare accepted but ignored.
Illama parses Hermes-3 style tool calls and <think>...</think> reasoning
blocks out of the raw model output and surfaces them as structured fields:
| Surface | Tools | Reasoning |
|---|---|---|
/v1/chat/completions |
choices[0].message.tool_calls[] (OpenAI shape) + finish_reason: "tool_calls" |
choices[0].message.reasoning_content |
/api/chat |
message.tool_calls[] (Ollama shape: no id/type/index, arguments is an object) |
message.thinking |
You pass tools the same way you would to OpenAI / Ollama:
curl -s http://localhost:11434/v1/chat/completions -d '{
"model": "Qwen2.5-7B-Instruct",
"messages": [{"role":"user","content":"What is the weather in Tokyo?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get the current weather for a city.",
"parameters": {"type":"object","properties":{"city":{"type":"string"}}}
}
}]
}'The tool list is plumbed through the tokenizer's chat template
(apply_chat_template(messages, tools=...)), so Hermes-trained checkpoints
(Qwen 2.5/3, Hermes-2-Pro, DeepSeek-R1) see the schema in their prompt and
emit the call in the format the parser recognises. This is the same job
OpenVINO Model Server does with --tool_parser hermes3 --reasoning_parser qwen3
— illama just does it in Python so OVMS isn't required.
Models that don't natively emit <tool_call> tags will simply produce
text answers; you get the same finish_reason: "stop" as today and no
tool_calls field. Streaming on both /v1/* and /api/* emits one chunk
per tool call as soon as </tool_call> closes.
Illama speaks both OpenAI (/v1/...) and Ollama (/api/...) APIs. Point Open WebUI at whichever surface you prefer. The default port is 11434. If that port is already taken (for example by Ollama), set ILLAMA_PORT in the environment or in deploy/.env.
On a machine with both an Intel iGPU and an Arc dGPU, set ILLAMA_DEVICE=GPU.1 so Illama uses the discrete GPU. GPU alone selects GPU.0.
| Mode | Setting | Example |
|---|---|---|
| Ollama | Ollama Base URL | http://127.0.0.1:11434 |
| OpenAI-compatible | API Base URL | http://127.0.0.1:11434/v1 |
| API Key | — | empty, or any string if auth is disabled; match ILLAMA_API_KEY if set |
-
Start illama:
illama serve # or: cd deploy && docker compose up -d -
In Open WebUI Admin → Connections, add either:
- Ollama:
http://<host>:11434, or - OpenAI:
http://<host>:11434/v1with any API key if auth is off
- Ollama:
-
Save. Models from
/opt/illama/modelsappear in the picker.
curl -s http://127.0.0.1:11434/api/tags | python3 -m json.tool | head
curl -s http://127.0.0.1:11434/v1/models | python3 -m json.tool | head
curl -s http://127.0.0.1:11434/illama/health | python3 -c \
'import sys,json; d=json.load(sys.stdin); print(d["device"], d["status"])'Important:
GET /v1returning404is normal. OpenAI clients still usehttp://<host>:11434/v1because they call/v1/modelsand/v1/chat/completions.
If auth is enabled, put the same bearer token value into the OpenWebUI API Key field.
| Requirement | Minimum | Recommended |
|---|---|---|
| OS | Ubuntu 22.04 | Ubuntu 24.04+ |
| Docker | 20.10+ | 24.0+ |
| Python | 3.9+ | 3.11+ |
| RAM | 16 GB | 32 GB |
| VRAM (GPU) | 8 GB | 16 GB |
| Disk | 50 GB free | 100 GB+ SSD |
- Intel Arc GPUs: A770, A750, A580, B580, etc.
- Intel Integrated GPUs: Iris Xe, UHD
- CPU-only: Works but slower
# Add Intel graphics PPA
sudo add-apt-repository -y ppa:kobuk-team/intel-graphics
sudo apt update
# Install drivers
sudo apt install -y libze-intel-gpu1 libze1 intel-opencl-icd
# Add user to video group
sudo usermod -aG video $USER
sudo usermod -aG render $USER
# Reboot
sudo reboot
# Verify
clinfo | head -20# Check if port is in use
sudo lsof -i :11434
# Check Docker logs
docker logs illama-service
# Run with verbose mode
illama -v serveThe model files are valid but the installed OpenVINO GenAI version doesn't yet know the model's architecture. Common message: Unsupported '<arch>' VLM model type (or similar for any new model_type that landed in master after the pinned release).
Two fixes:
-
Switch to a supported model for now. The OpenVINO GenAI release notes track newly-added VLMs. Stable
openvino-genai==2026.4.0.0includes the types that landed through 2026.4, includinggemma4(OpenVINO/gemma-4-E4B-it-int4-ov). -
Bump OpenVINO GenAI using How to update runtime libraries.
openvino,openvino-genai, andopenvino-tokenizersmove together. Stable2026.4.1/2026.4.1.0is the current target and includes Gemma-4.
Unlike a GPU-state error, this error does not restart the container — the runner keeps serving other models. Inspect the actual reason returned in the API response or illama logs.
Illama exposes a lightweight Prometheus-style metrics endpoint:
curl http://localhost:${ILLAMA_PORT:-11434}/metricsIt includes:
- total requests and errors
- chat vs completion request counters
- streaming vs non-streaming request counters
- auth and validation failure counters
- active generations
- model load/unload counters
- model load time totals
- prompt/completion token totals
# Check HuggingFace authentication
huggingface-cli whoami
# Login if needed
huggingface-cli login
# Check disk space
df -h /opt/illamaSymptoms:
- VRAM is near full
- GPU utilization is high
- the request eventually returns, but there is no visible partial output
This is usually expected behavior for slow generation, not a crash. Streaming is now supported, but if a client disables streaming or buffers the response you may still see long waits before output appears.
For large models such as gemma-3-27b-it on Battlemage/B50-class GPUs:
- small replies may return in under a second
- medium/long replies can take 30-60s+ before the full response finishes
- around
8-9 tokens/secis a realistic observed range for longgemma-3-27b-itreplies on this setup
Quick sanity check:
curl -s http://localhost:${ILLAMA_PORT:-11434}/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-3-27b-it",
"messages": [{"role":"user","content":"Reply with exactly: hello"}],
"max_tokens": 16,
"stream": false
}'If short prompts succeed but long prompts feel stalled, prefer:
- smaller explicit
max_tokens - smaller/faster models such as
gemma-3-12b-it-qat-q4_0-unquantized,Qwen3-14B-int4-ov, orDeepSeek-R1-Distill-Qwen-14B-int4-ov - using streaming-capable clients for very large models so users can see progress immediately
If a model fails during load, behavior is controlled by load_failure_policy:
restart: default, exits the process so Docker can restart cleanlyreturn_error: keeps the server alive and returns an API error
Example:
{
"load_failure_policy": "return_error"
}For quick testing on a live server, you can override the config with an environment variable:
export ILLAMA_LOAD_FAILURE_POLICY=return_error
illama stop
illama serve# Check device exists
ls -la /dev/dri
# Check OpenCL
clinfo
# Verify user is in video group
groups | grep videoLibraries are baked into the images at docker build. Restarting illama-service does not change them. The next upgrade starts by reading the pip install lines in the Dockerfiles that Docker actually builds, then replacing those exact version strings.
This is the target until a newer release is confirmed on PyPI. Update this table when that happens, then follow the procedure below.
| Package | Latest stable | Where it is pinned |
|---|---|---|
openvino |
2026.4.1 |
runner and builder pip install (patch on the 2026-09-16 2026.4 line) |
openvino-genai |
2026.4.1.0 |
same line as openvino; move them together |
openvino-tokenizers |
2026.4.1.0 |
pulled in by openvino-genai when the Dockerfile has no explicit pin |
optimum-intel |
2.2.0 |
builder only; transformers must stay >=4.51,<5.6 |
transformers |
4.57.6 |
builder only; inside optimum-intel's >=4.51,<5.6 window |
fastapi / uvicorn / pydantic |
0.136.1 / 0.46.0 / 2.13.3 |
runner HTTP stack; independent of the OpenVINO bump |
openvino, openvino-genai, and openvino-tokenizers are one build. Mixing release families breaks the binary ABI.
Release notes: OpenVINO 2026.4, optimum-intel 2.2.0.
runner/Dockerfile and builder/Dockerfile pin those stable releases directly. ./setup.sh copies them onto /opt/illama and rebuilds. Checked against PyPI on 2026-10-13: OpenVINO 2026.4.1 is still the latest stable. Do not install a newer transformers than optimum-intel allows (<5.6).
Run this on the host that runs Illama. Read the files before editing. Replace the version strings you actually see; the sed below is an example (2026.4.0 → 2026.4.1).
1. See what is running, and the lines you will replace
docker ps --filter name=illama-service --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
docker exec illama-service python3 -c "import importlib.metadata as m
for p in ['openvino','openvino-genai','openvino-tokenizers','optimum-intel','fastapi','uvicorn','pydantic']:
try:
print(f'{p}=={m.version(p)}')
except m.PackageNotFoundError:
print(f'{p} not installed')"
grep -nE 'pip install|openvino|optimum-intel|extra-index-url|--pre' \
/opt/illama/runner/Dockerfile /opt/illama/builder/DockerfileWrite down the host port from docker ps. illama serve publishes 11434 unless ILLAMA_PORT is set.
2. Replace those exact pins
openvino-genai is replaced before openvino so the shorter name cannot match the longer one. Adjust the old version strings to match the grep output.
sed -i \
-e 's/"openvino-genai==2026.4.0.0"/"openvino-genai==2026.4.1.0"/' \
-e 's/"openvino==2026.4.0"/"openvino==2026.4.1"/' \
/opt/illama/runner/Dockerfile /opt/illama/builder/DockerfileIf a runner line still has the nightly index, drop that too:
sed -i \
-e '/^RUN pip install/s/ --pre//' \
-e '/wheels\/nightly/d' \
/opt/illama/runner/DockerfileConfirm before building:
grep -nE 'openvino|optimum-intel' /opt/illama/runner/Dockerfile /opt/illama/builder/Dockerfile3. Rebuild and start on the same host port
VER="$(tr -d '[:space:]' < /opt/illama/VERSION 2>/dev/null || echo dev)"
docker build --build-arg "ILLAMA_VERSION=${VER}" \
-t illama-runner:latest -t "illama-runner:${VER}" \
/opt/illama/runner
docker build --build-arg "ILLAMA_VERSION=${VER}" \
-t illama-builder:latest -t "illama-builder:${VER}" \
/opt/illama/builder
export ILLAMA_PORT="${ILLAMA_PORT:-11434}"
illama stop || true
illama serve
docker exec illama-service python3 -c "import importlib.metadata as m
for p in ['openvino','openvino-genai','openvino-tokenizers']:
print(f'{p}=={m.version(p)}')"Expected after the 2026-10-04 bump: openvino==2026.4.1, openvino-genai==2026.4.1.0, openvino-tokenizers==2026.4.1.0. Models in /opt/illama/models stay. The first load of each model recompiles into /opt/illama/cache.
The runner build changes the live server. The builder build is the large one (optimum-intel pulls PyTorch) and matters on the next illama pull that still needs conversion.
When a newer git tree is supposed to carry these pins, update the Dockerfiles in the checkout first, then ./setup.sh so it copies them onto /opt/illama and rebuilds the runner. Build the builder image with the docker build above if conversions need the new optimum-intel as well. Keep ILLAMA_PORT at the port docker ps showed.
VLM models such as OpenVINO/gemma-4-E4B-it-int4-ov (Image-Text-to-Text) are loaded automatically when the model directory contains openvino_language_model.xml instead of openvino_model.xml. Both the OpenAI and the Ollama API surfaces accept image inputs.
Pass content as the multimodal content-array form. Both data: URLs (base64) and https:// URLs are accepted. Decoding budget per image is 20 MB; HTTPS fetch timeout is 10 seconds.
# Base64 data URL
curl -s http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-4-E4B-it-int4-ov",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "What is in this image?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
]
}],
"max_tokens": 256,
"temperature": 0.2
}'
# Direct HTTPS URL
curl -s http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-4-E4B-it-int4-ov",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe this picture."},
{"type": "image_url", "image_url": {"url": "https://example.com/cat.jpg"}}
]
}]
}'Use the images field with raw base64 strings (no data: prefix).
curl -s http://localhost:11434/api/chat \
-H "Content-Type: application/json" \
-d '{
"model": "gemma-4-E4B-it-int4-ov",
"messages": [{
"role": "user",
"content": "What do you see?",
"images": ["iVBORw0KGgo..."]
}],
"stream": false
}'/api/generate accepts images on the request body itself (single-prompt form).
temperature, top_p, max_tokens (and Ollama's num_predict) are honored on VLM models the same way as on plain LLM models.
These are tracked items where the current implementation is intentionally narrow or has known sharp edges. None of them block typical OpenWebUI / Ollama-client / OpenAI-client usage but they are worth knowing about when planning deployments.
response_formatis best-effort. OpenVINO GenAI does not yet expose constrained-grammar sampling.response_format={"type":"json_object"}is implemented as a prepended system instruction, so model adherence depends on the model.response_format={"type":"json_schema"}is not implemented.presence_penaltyandnare accepted but ignored.frequency_penalty > 0is mapped onto OpenVINO'srepetition_penaltyas1 + frequency_penalty, which is a best-effort translation, not a one-to-one.- Single in-flight model. This is by design - one model loaded at a time, sequential generation. There is no batching across concurrent requests.
- Image fetch is unauthenticated. When an OpenAI
image_urlpoints at an HTTPS URL, the runner does a plainrequests.getwith a 10s timeout and 20 MB cap. There is no allowlist; if you need to restrict outbound access, run the runner container in a network namespace that scopes egress.
Illama is built on OpenVINO GenAI. Alternative inference backends for Intel Arc / Battlemage exist but each has trade-offs:
| Backend | Status (September 2026) | When to consider |
|---|---|---|
| OpenVINO GenAI 2026.4 | Current stable. First-class Battlemage support, Gemma-4 VLMs, wider model coverage than 2026.1. | Default. Bump through How to update runtime libraries. |
llama.cpp (SYCL or Vulkan) |
Active. Major Battlemage optimizations through 2026 (e.g. ggml-org/llama.cpp PRs #21527/#21638/#22066). Q4_K_M ~20 t/s on B70 for 27B models is realistic. |
Future migration target if specific quantizations or ggml-format models matter. Not a drop-in - the runner would need a rewrite. |
OpenVINO + llama.cpp backend |
New in 2026.1: OpenVINO ships a llama.cpp backend that targets Intel GPUs. Lets you run ggml/gguf models through OpenVINO's runtime. |
Worth evaluating once it stabilises; could give us ggml models without leaving the OpenVINO ecosystem. |
| IPEX-LLM | Archived by Intel on 2026-01-28. No further updates. | Do not migrate to IPEX-LLM. Anything you build on it will be unsupported within the year. |
| vLLM | NVIDIA-first. Intel paths are CPU-only or via Habana/Gaudi. | Not relevant for Intel Arc / Battlemage GPU inference. |
The pragmatic recommendation for this hardware is: stay on OpenVINO GenAI, currently stable 2026.4.1, and revisit another backend only when OpenVINO shows a specific perf or model-coverage gap.
Stability and CLI ergonomics ahead of feature breadth:
- harden SSE / NDJSON streaming against client disconnect (drain-on-cancel for the worker thread)
- enforce
allow_streamon the Ollama/api/*surface so the global toggle actually applies to all clients - bound
/metricsper-model cardinality - harden
illama pullagainst partial-download poisoning andillama rmagainst path-traversal of the model name
- openvino_build_deploy: Pre-built samples and reference kits for OpenVINO applications. Contains:
- AI Reference Kits (production-ready examples)
- Interactive demos
- Training notebooks for GenAI and LLMs
- Useful for learning but not directly applicable to Illama's architecture
Illama is Apache-2.0. It runs on top of these projects. Their licenses stay with them; this table is attribution, not a relicense.
| Stack | Role here | License |
|---|---|---|
| OpenVINO, OpenVINO GenAI, OpenVINO Tokenizers | Inference runtime in the runner image | Apache-2.0 |
| Intel Graphics Compute Runtime / Level Zero | GPU driver userspace inside the runner image | MIT |
| optimum-intel | Model export (optimum-cli export openvino) in the builder image |
Apache-2.0 |
| Transformers, Accelerate | Export-time model code in the builder image | Apache-2.0 |
| NNCF | Pulled in by optimum-intel[openvino] for INT4 compression |
Apache-2.0 |
| FastAPI | HTTP API | MIT |
| Uvicorn | ASGI server | BSD-3-Clause |
| Pydantic | Request and response models | MIT |
| Ubuntu 24.04 | Base image | Various; see Canonical's terms |
Model weights you illama pull keep the license of that checkpoint (some require accepting terms on Hugging Face). Illama does not grant rights to those weights.