LLM inference and serving on T-Head Zhenwu AI accelerators
Documentation | Installation | Supported Features | Contributing
vLLM SAIL is a hardware plugin for vLLM. It connects vLLM's inference engine to the SAIL software stack, bringing PPU kernels, attention backends, quantization and model integration to the standard vLLM Python API and OpenAI-compatible server.
SAIL is the software brand and Zhenwu (真武) is the hardware product
name. The T-Head AI chips are Zhenwu 810E and Zhenwu M890. The Python
distribution is vllm-sail, the import package is vllm_sail, and the vLLM
plugin name is ppu.
The plugin builds native kernels with HGGC, using the hgcc compiler and
cmake-hgcc. PPU libraries and
Triton for SAIL provide specialized
compute paths. vLLM discovers the installed plugin through its platform and
general plugin entry points.
- Native PPU kernels: activation, normalization, quantization, cache operations, sampling and MoE routing compiled with HGGC.
- Attention: PPU FlashAttention integration, MLA and sparse attention, plus GDN and KDA linear attention paths.
- Dense and MoE compute: DeepGEMM, ACEXT and Triton backends, with BF16/FP16 and selected INT8, FP8, MXFP4 and INT4 quantization paths.
- Model integration: PPU model registrations and adaptations for dense, mixture-of-experts and hybrid architectures.
- Developer tooling: tests that run without a PPU, reproducible kernel generation for upstream sources, patch checks and device correctness tests.
Availability depends on the device, SDK, backend and checkpoint format. See the feature matrix for implementation scope and constraints, and the verification guide for validating a deployment.
| Component | Requirement |
|---|---|
| Hardware | Zhenwu 810E and Zhenwu M890 AI chips; build targets ppu_10 and ppu_15 |
| Operating system | Linux for PPU builds and inference |
| Python | 3.10–3.13; choose a version supported by your SDK packages |
| vLLM | >=0.31.0,<0.32.0, built with VLLM_TARGET_DEVICE=empty |
| SAIL software | SAIL SDK, SAIL PyTorch and matching PPU compute libraries |
| Build tools | hgcc, CMake, Ninja and a C++ compiler; CMake fetches pinned cmake-hgcc sources when needed |
The generated upstream kernel sources are pinned to vLLM v0.31.0. The installation guide describes dependency setup and native extension checks.
Prepare the SAIL environment, then clone and install the plugin:
git clone https://github.com/t-head/vllm-sail.git
cd vllm-sail
python -m pip install -r requirements/build.txt -r requirements/ppu.txt
python -m pip install --no-build-isolation --no-deps -e .
python -c "from vllm_sail import collect_env; collect_env()"After the installation checks pass, start a server with a local model:
vllm serve /path/to/model \
--served-model-name ppu-model \
--dtype bfloat16 \
--max-model-len 4096 \
--host 127.0.0.1 --port 8000Follow the quickstart for a client request, offline inference and model configuration.
| Guide | Contents |
|---|---|
| Installation | SDK prerequisites, source installation and troubleshooting |
| Quickstart | Online serving and offline inference |
| Supported features | Backends, quantization, model integration and limitations |
| Configuration | Backend selection and PPU environment variables |
| Verification | Environment, kernel and model validation |
| Architecture | Plugin lifecycle and source layout |
| HGGC kernels | Native builds, source ownership and regeneration |
| Kernel benchmarks | Device benchmarks and MoE tuning tools |
Contributions are welcome: report a reproducible issue, improve documentation, add a test, tune a PPU workload or implement a kernel. Many development tasks can be done without PPU hardware.
Start with CONTRIBUTING.md for the development environment, test commands and pull request workflow. Use this repository's Issues to discuss bugs and proposed features, and include the environment and validation details requested by the templates.
vLLM SAIL is licensed under the Apache License 2.0. Third-party source files retain their original copyright and license notices.