Skip to content

quickstart.md

Latest commit

 

History

History
63 lines (41 loc) · 2.71 KB

quickstart.md

File metadata and controls

63 lines (41 loc) · 2.71 KB
Raw
title Quickstart
description Set up llm-helper, pull a model, and serve it.

This guide takes a machine from nothing to serving a model through vLLM's OpenAI-compatible endpoint. It assumes vLLM is installed, usually in a venv, and that the llm-helper release for your platform is on the machine.

Run setup

  1. Run llm-helper setup.

Setup detects your hardware and any existing vLLM install, generates an API key, and writes config. Each model you start later runs as its own service — a systemd unit on Linux, a launchd job on macOS. It never overwrites existing config and never restarts a running server.

  1. If vllm isn't on your PATH, set VLLM_BIN in ~/.config/vllm-helper/config.env to its full path.

Pull a model

  1. Run llm-helper new with a HuggingFace repo:

    llm-helper new Qwen/Qwen3-32B

This pulls the model, saves its settings as a profile, and starts serving it. The weights are downloaded by vLLM as it starts, so the first start takes as long as the download does.

To browse instead of naming a repo outright, run llm-helper search <query>. It lists matches with each one's weight size checked against what this machine can hold, and walks you through pulling the one you pick.

Check it

  1. Run llm-helper status.

status reports each running model, its port, and whether its endpoint responds. It distinguishes a load that is still in progress from a service failing to start — a large model can take many minutes, and the two look identical from the outside.

Point a client at it

Any OpenAI-compatible client can use the endpoint at http://<machine>:8001/v1, the first model's port, with the API key from ~/.config/vllm-helper/secrets.env.

Run another model

  1. Run llm-helper list to see the saved profiles.
  2. Run llm-helper up <profile> to serve another one beside it, on the next free port.
  3. Run llm-helper down <profile> to stop one.

Starting a model takes a few minutes when the weights are cached and longer on first download. Several models at once covers ports, memory, and offering every model through one port.

Script it

  • status, list, search, inspect, doctor and recipes list take --json.
  • up, new and restart take --wait to block until the endpoint is healthy, --no-follow to return immediately, and --timeout.
  • When stdout is piped, commands don't tail logs or start pickers. When stdin isn't a terminal, prompts take their defaults and sudo runs with -n.

Next steps

  • See what a profile holds in Profiles.
  • Manage the machine from a browser with the Web panel.
  • Serve with mlx-lm or llama.cpp instead of vLLM with Providers.