llm-wake-proxy

⚠️ Alpha quality / personal project. This works for my setup but has rough edges, minimal error handling in places, and no stability guarantees. Not recommended for production or any environment where reliability matters.

A standalone Rust proxy that keeps a private llama.cpp model host asleep when idle and wakes it on demand. Exposes an OpenAI-compatible API so standard clients like opencode work without custom glue.

Architecture

Clients (opencode, etc.)
    |
    v
llm-wake-proxy (Kubernetes)
    |  - Request admission + queueing
    |  - Cold-start orchestration
    |  - SSH tunnel lifecycle
    |
    v
SSH Tunnel --> llama-server (loopback on bare-metal host)

The proxy owns the OpenAI-compatible API, cold-start orchestration, and SSH tunnel lifecycle. The bare-metal host runs llama-server and an inhibit holder under systemd --user only when needed.

Configuration

All configuration is via environment variables.

Proxy

Variable	Default	Description
`PORT`	`3000`	Proxy listen port
`MODEL_ALIAS`	`llm-wake-proxy`	Stable model name exposed to clients
`MODEL_OWNED_BY`	`llm-wake-proxy`	Owner string for `/v1/models`
`MODEL_PROVIDER_ID`	`llama.cpp`	Provider ID for `/v1/models`
`EMBEDDINGS_ENABLED`	`true`	Enable embeddings forwarding
`EMBEDDINGS_MODEL_ALIAS`	`{MODEL_ALIAS}-embeddings`	Model alias for the dedicated embeddings backend (dual-backend mode, see below)
`EMBEDDINGS_MODEL_OWNED_BY`	`MODEL_OWNED_BY`	Owner string for the embeddings model in `/v1/models` (dual-backend mode)
`EMBEDDINGS_MODEL_PROVIDER_ID`	`MODEL_PROVIDER_ID`	Provider ID for the embeddings model in `/v1/models` (dual-backend mode)
`WARM_MAX_ACTIVE_REQUESTS`	`2`	Max concurrent upstream requests
`WARM_MAX_QUEUED_REQUESTS`	`16`	Max queued warm requests (0 = no queue)
`WARM_QUEUE_TIMEOUT_SECS`	`30`	Max seconds a request waits in queue
`COLD_START_MAX_WAITING`	`32`	Max concurrent cold-start waiting requests

Host / SSH

Variable	Default	Description
`SSH_HOST`	(required)	Bare-metal host address (IP or Tailscale hostname)
`SSH_USER`	(required)	SSH user on the host
`SSH_PORT`	`22`	SSH port
`HELPER_PATH`	`/usr/local/bin/llm-wake-proxy-helper`	Path to helper binary on host
`MODEL_PATH`	(required)	Path to model file on host (for model verification)
`TUNNEL_LOCAL_PORT`	`18080`	Local port for SSH tunnel
`LLAMA_SERVER_PORT`	`8080`	Remote port on host (llama-server)
`EMBEDDINGS_MODEL_PATH`	(unset)	Path to a dedicated embeddings model file on host. Setting this activates dual-backend mode: chat and embeddings run as separate `llama-server` processes with independent lifecycle tracking. Implies `EMBEDDINGS_ENABLED=true`, overriding any explicit `EMBEDDINGS_ENABLED=false`
`EMBEDDINGS_TUNNEL_LOCAL_PORT`	`18081`	Local port for the embeddings SSH tunnel (dual-backend mode)
`EMBEDDINGS_LLAMA_SERVER_PORT`	`8081`	Remote port on host for the embeddings `llama-server` (dual-backend mode)

Wake-on-LAN

Variable	Default	Description
`WOL_MAC_ADDRESS`	(required)	MAC address of host (colon-separated)
`WOL_BROADCAST_ADDR`	`255.255.255.255`	Broadcast address
`WOL_PORT`	`9`	WOL UDP port

Host Setup

Prerequisites

llama-server installed and accessible on the host
SSH key-based auth (no password) from the proxy's service account to the host
Host SSH host key already accepted (StrictHostKeyChecking=accept-new)
systemd --user managing llama-server and the inhibit holder

Helper Binary

The helper binary (llm-wake-proxy-helper) runs on the host over SSH. It provides:

# Machine-readable host status
llm-wake-proxy-helper status

# Ensure llama-server is started (idempotent, singleton)
EXPECTED_MODEL_PATH=/models/model.gguf llm-wake-proxy-helper ensure-started chat default

# Dual-backend mode: same command, targeting the embeddings server
EXPECTED_MODEL_PATH=/models/embed-model.gguf llm-wake-proxy-helper ensure-started embeddings embed-default

# Lease management
llm-wake-proxy-helper lease acquire --ttl 3600
llm-wake-proxy-helper lease release
llm-wake-proxy-helper lease inspect

All subcommands emit JSON on stdout and reserve stderr for human diagnostics.

systemd Units

The helper manages user-level systemd units:

llama-server.service — A static unit you create once. The helper starts it on demand via systemctl --user start llama-server. It should be disabled (not enabled for auto-start) so the host can sleep when idle.
llama-server-embeddings.service — Optional second static unit for dual-backend mode, used when the proxy is configured with EMBEDDINGS_MODEL_PATH. Same shape as llama-server.service but runs a dedicated embeddings model on its own port (default 8081). Also kept disabled. Override the unit name/port the helper looks for via LLAMA_SERVER_EMBEDDINGS_UNIT / LLAMA_SERVER_EMBEDDINGS_PORT if you deviate from the defaults.
llm-wake-proxy-inhibit — A transient unit created by systemd-run when the proxy acquires a lease. It runs systemd-inhibit --what=sleep to keep the host awake while requests are active. The helper removes it when the lease is released or expires.

Installing the llama-server unit

Copy llama-server.service from this repo to ~/.config/systemd/user/ on the host and edit the ExecStart line to point at your model and llama-server binary:

mkdir -p ~/.config/systemd/user
cp llama-server.service ~/.config/systemd/user/
# Edit ExecStart to match your setup

Then reload and disable it (so it only starts on-demand):

systemctl --user daemon-reload
systemctl --user disable llama-server.service

For dual-backend mode, repeat the same steps with llama-server-embeddings.service, pointing ExecStart at your embeddings model and --port 8081 (or whatever you set LLAMA_SERVER_EMBEDDINGS_PORT/EMBEDDINGS_LLAMA_SERVER_PORT to).

If the host needs to work while nobody is logged in, enable lingering for the user:

sudo loginctl enable-linger "$USER"

Deployment (Kubernetes)

A multi-stage Dockerfile and Helm chart ship in this repo.

Build and push the image

# One-time: log in to your registry (example uses GHCR).
echo $GITHUB_TOKEN | docker login ghcr.io -u <your-user> --password-stdin

# Build and push. Override IMAGE / TAG to taste.
make build IMAGE=ghcr.io/<your-user>/llm-wake-proxy TAG=0.1.0
make push  IMAGE=ghcr.io/<your-user>/llm-wake-proxy TAG=0.1.0

# Or multi-arch in one shot (requires buildx):
make buildx IMAGE=ghcr.io/<your-user>/llm-wake-proxy TAG=0.1.0

The image is a distroless cc-debian12:nonroot base with both llm-wake-proxy and llm-wake-proxy-helper plus the OpenSSH client (for the tunnel and helper RPC). It runs as UID 65532 with all capabilities dropped, a read-only root filesystem, and seccompProfile: RuntimeDefault.

Note: The chart defaults to hostNetwork: true because Wake-on-LAN requires sending a UDP broadcast from the node’s physical network interface. Standard pod networking isolates the broadcast, so the magic packet never reaches the LAN.

Provide the SSH key

The proxy opens an SSH tunnel to the bare-metal host, so the pod needs a private key. Create the Secret out of band so the key never lands in git:

ssh-keygen -t ed25519 -N '' -f ~/.ssh/llm-wake-proxy
ssh-keyscan -H llama.tailnet.example > known_hosts

kubectl create namespace llm-wake-proxy

kubectl create secret generic llm-wake-proxy-ssh-key \
  --namespace llm-wake-proxy \
  --from-file=ssh-privatekey=~/.ssh/llm-wake-proxy \
  --from-file=known_hosts=known_hosts

Authorise the public key on the bare-metal host as usual (e.g. ~/.ssh/authorized_keys).

Install the chart

helm install llm-wake-proxy ./charts/llm-wake-proxy \
  --namespace llm-wake-proxy \
  --set ssh.host=llama.tailnet.example \
  --set ssh.user=jon \
  --set wol.macAddress=AA:BB:CC:DD:EE:FF \
  --set ssh.modelPath=/models/qwen2.5-7b-instruct-q4_k_m.gguf \
  --set proxy.modelAlias=qwen2.5-7b-instruct \
  --set ssh.existingSecret=llm-wake-proxy-ssh-key \
  --set persistence.ssh-key.enabled=true \
  --set controllers.main.containers.main.image.repository=ghcr.io/jonmast/llm-wake-proxy

Required values: ssh.host, ssh.user, ssh.modelPath, wol.macAddress. The chart enforces these with required and helm install will refuse to proceed without them.

Verify

kubectl --namespace llm-wake-proxy port-forward \
  svc/llm-wake-proxy 8080:3000

curl -s http://localhost:8080/healthz
# {"status":"ok"}

curl -s http://localhost:8080/status | jq

A cold host will return 503 warming_up from /v1/chat/completions with a Retry-After header until WOL, SSH, and llama-server are all ready. See Verification below for the full status contract.

Chart values

The full list lives in charts/llm-wake-proxy/values.yaml. Highlights:

Value	Default	Notes
`replicaCount`	`1`	V1 keeps coordination state in memory. No HA.
`service.type`	`ClusterIP`	Use `LoadBalancer`/`NodePort` to expose externally.
`resources.requests/limits`	100m/128Mi → 1000m/512Mi	Tune for your model.
`probes.liveness/readiness`	enabled	Both hit `/healthz`.
`ssh.mountPath`	`/home/nonroot/.ssh`	Override if you ship a different layout.
`proxy.extraEnv`	`[]`	Merge arbitrary `env:` entries (e.g. `RUST_LOG=info`).

Other useful targets

make lint        # cargo check + clippy + helm lint
make render      # helm template dry-run with sane defaults
make package     # helm package the chart into dist/
make uninstall   # helm uninstall llm-wake-proxy

Verification

Health

curl http://localhost:8080/healthz
# {"status":"ok"}

Status

curl http://localhost:8080/status

Returns:

{
  "chat": {
    "model_alias": "llm-wake-proxy",
    "state": "ready",
    "capability": "ready",
    "capability_reason": null,
    "tunnel": "ready",
    "last_wake_attempt_at": 1717156800,
    "lease_expires_at": 1717158600,
    "host_unit": {
      "llama_server_unit": "active",
      "inhibit_unit": "activating"
    }
  },
  "embeddings": {
    "model_alias": "llm-wake-proxy-embeddings",
    "state": "ready",
    "capability": "ready",
    "capability_reason": null,
    "tunnel": "ready",
    "last_wake_attempt_at": 1717156800,
    "lease_expires_at": 1717158600,
    "host_unit": {
      "llama_server_unit": "active",
      "inhibit_unit": "activating"
    }
  },
  "metrics": {
    "cold_starts": 1,
    "warm_requests": 42,
    "queue_full_rejections": 0,
    "queue_timeouts": 0,
    "wake_attempts": 1,
    "wake_failures": 0,
    "tunnel_drops": 0,
    "embeddings_degraded": 0,
    "forwarding_errors": 0,
    "chat_requests": 42,
    "embeddings_requests": 5
  }
}

The embeddings block is null unless EMBEDDINGS_ENABLED=true. In the default (shared-backend) configuration, chat and embeddings reflect the same underlying llama-server process; in dual-backend mode (EMBEDDINGS_MODEL_PATH set) they track two independent processes with their own state, tunnel, and host units.

State Transitions

State	Meaning
`cold`	Backend has never been probed or needs a fresh wake
`warming`	Wake sent, waiting for SSH + helper
`ready`	Backend is live and tunnel is established
`error`	Something failed (SSH, wake, helper, tunnel)

Error Semantics

Status	Type	Meaning
`503`	`warming_up`	Backend is starting; retry after `Retry-After`
`503`	`backend_error`	Backend observation failed
`503`	`backend_unavailable`	Backend ready but forwarding failed
`429`	`overloaded`	Warm execution queue is full or timed out
`400`	`invalid_request_error`	Bad JSON, unsupported fields, bad model
`400`	`unsupported_role`	Unsupported message role in chat request
`400`	`model_not_found`	Requested model doesn't match configured alias
`400`	`unsupported_embeddings`	Embeddings disabled or degraded
`503`	`upstream_error`	Upstream backend returned an error
`503`	`request_cancelled`	Request cancelled (lease timeout, etc.)

Metrics

The /status endpoint includes a metrics object with atomic counters:

cold_starts - Number of cold-start transitions
warm_requests - Requests served via warm path
queue_full_rejections - Requests rejected (queue full)
queue_timeouts - Requests that timed out in queue
wake_attempts - WOL packets sent
wake_failures - WOL/SSH failures during wake
tunnel_drops - SSH tunnel disconnections
embeddings_degraded - Embeddings degraded transitions
forwarding_errors - Upstream forwarding errors
chat_requests - Total chat requests received
embeddings_requests - Total embeddings requests received

Lifecycle Timing

Variable	Default	Description
`COLD_WAIT_BUDGET_SECS`	`90`	How long a cold request waits before returning 503
`HARD_BOOT_DEADLINE_SECS`	`300`	Maximum time from first wake to backend ready
`BOOTSTRAP_POLL_INTERVAL_MS`	`1000`	Polling interval during bootstrap
`RETRY_AFTER_SECS`	`10`	Value for Retry-After header in warming responses

Design Constraints

Single replica: V1 keeps coordination state in memory. No HA.
No auth: The proxy accepts Authorization headers but does not enforce auth. Use network-level access control.
Private LAN only: Not intended for public internet exposure.
No root/sudo: The service and helper run as unprivileged user.
Host stays loopback-only: All inference traffic routes through the SSH tunnel.

Name		Name	Last commit message	Last commit date
.cargo		.cargo
.github/workflows		.github/workflows
charts/llm-wake-proxy		charts/llm-wake-proxy
docs/agents		docs/agents
src		src
.dockerignore		.dockerignore
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
CONTEXT.md		CONTEXT.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
Makefile		Makefile
README.md		README.md
llama-server-embeddings.service		llama-server-embeddings.service
llama-server.service		llama-server.service

Navigation Menu

Repository files navigation

llm-wake-proxy

Architecture

Configuration

Proxy

Host / SSH

Wake-on-LAN

Host Setup

Prerequisites

Helper Binary

systemd Units

Installing the llama-server unit

Deployment (Kubernetes)

Build and push the image

Provide the SSH key

Install the chart

Verify

Chart values

Other useful targets

Verification

Health

Status

State Transitions

Error Semantics

Metrics

Lifecycle Timing

Design Constraints

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

llm-wake-proxy

Architecture

Configuration

Proxy

Host / SSH

Wake-on-LAN

Host Setup

Prerequisites

Helper Binary

systemd Units

Installing the llama-server unit

Deployment (Kubernetes)

Build and push the image

Provide the SSH key

Install the chart

Verify

Chart values

Other useful targets

Verification

Health

Status

State Transitions

Error Semantics

Metrics

Lifecycle Timing

Design Constraints

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages