Zero-trust lifecycle automation for bare-metal AI infrastructure
Documentation | Quick Start | Local Development | Architecture | REST API Reference | Contributing
Repository move notice: On September 4, 2026, the NICo repository moved from the NVIDIA GitHub organization to
dsx-ai-factory/infra-controller. Existing repository URLs and standard Git operations continue to work through GitHub redirects. If you maintain automation or integrations that referenceNVIDIA/infra-controller, such as GitHub Actions, webhooks, or pinned repository URLs, update them todsx-ai-factory/infra-controller.
NICo delivers lifecycle automation for bare-metal systems that secures datacenter infrastructure at its foundation. It is an open source suite of microservices that run next to the hardware, managing and automating the full bare-metal lifecycle: hardware discovery, firmware validation, BlueField DPU provisioning, network isolation, tenant provisioning, and secure sanitization between tenants. NVIDIA Cloud Partners (NCPs) and infrastructure operators can use NICo to stand up and operate AI factory-scale infrastructure through APIs instead of runbooks.
Full component diagram: every Site Controller service, host agent, and off-the-shelf dependency
AI factories need rack-level management, host lifecycle automation, and workload isolation that general-purpose tools do not provide as one integrated system. Without it, operators are left with manual discovery and firmware alignment on every rack bring-up, no single enforcement point for tenant isolation across Ethernet, InfiniBand, and NVLink, custom scripts for sanitization and attestation, and firmware drift across hardware generations.
NICo closes these gaps. In the default configuration, a managed host is a
server with one or more BlueField DPUs, and the DPU is the enforcement
boundary. NICo provisions and controls the DPU directly, independently of
whatever the tenant runs on the host, so the layers above it can treat bare
metal as a reliable building block. NICo also supports zero-DPU hosts, where
the DPU policy is ignore or nic. Their tenant instances attach to Flat
VPCs on the underlay, and the operator's fabric provides the isolation. Refer
to the glossary for both host types.
Discovers BMCs over the out-of-band network with Redfish, pairs DPUs to
hosts, validates each machine against its expected SKU, runs burn-in and
fabric tests, and enforces a UEFI and BMC firmware baseline before a host
is offered to tenants.
|
Installs the DPU OS, provisions Host-Based Networking (HBN) with
Containerized Cumulus, manages DPU BMC, NIC, UEFI, and ATF firmware, and
runs the DPU agent that continuously applies the desired network state.
|
Enforces per-tenant boundaries on every plane without touching physical
switches: VXLAN/EVPN and VRFs on Ethernet, UFM P_Key partitions on
InfiniBand, NMX-C partitions on NVLink, and Spectrum-X partitioning.
|
Treats every host as untrusted by default. When attestation is enabled,
verifies measured boot PCRs and TPM signatures. Locks down UEFI during
tenant use, manages BMC and UEFI credentials, and never relies on
in-band host reporting for security decisions.
|
Keeps the fleet on a site-wide firmware baseline. Operators declare the
expected host and DPU versions and, when automatic updates are enabled,
NICo detects drift and schedules out-of-band updates on healthy,
unoccupied hosts without disrupting active tenants.
|
Boots any iPXE-installable OS, allocates IP addresses, and configures
BGP, DHCP, and DNS. When a tenant leaves, erases NVMe and SAS/SATA
drives, verifies the UEFI memory-overwrite request, resets InfiniBand
devices, and tears down the tenant network before reuse.
|
For GB200 NVL72 systems, NICo Flow extends all of this to the rack. It treats racks and NVL domains as first-class entities and sequences power and firmware operations across trays. NICo also gates instance allocation on NVLink fabric health. Refer to Key Capabilities for the complete list and to the Hardware Compatibility List for supported servers and DPUs.
NICo REST authorizes every caller as a provider or a tenant, or as a service account that acts as both. All of them work through the same REST API, and every endpoint is documented in the REST API Reference, including the typical API call flows for each persona.
|
The NCP or infrastructure operator who owns the site. Providers register
sites, ingest and validate hardware, define tenants and instance types,
allocate hosts, manage racks, and run firmware and maintenance
operations.
Tools: the REST API and its nicocli client
for day-to-day operations, and the
nico-admin-cli
for site-administrator tasks on the Core gRPC API.
Start with: Ingesting Hosts (REST API), Tenant Management, and Firmware Updates. |
The organization that consumes bare-metal instances. Tenants manage
their own instances, SSH keys, and VPC peering inside the isolation
boundary the provider established, and they never see other tenants'
hardware or networks.
Tools: the REST API with tenant-scoped JWTs, the serial console through the SSH Console service, and the metadata service on each instance. Start with: the REST API Reference and the instance sections of Tenant Management. |
Service Account mode is the third option. NICo REST can be configured so that one organization holds both the provider and tenant roles through a service account. This suits deployments that manage tenancy in a system above NICo and call it as one principal. Refer to Tenant Management with Keycloak for the service-account client setup and to the authorization roles in the glossary for what each role family can do.
NICo organizes bare-metal management into three phases. Day 0 takes hardware from "racked and cabled" to "ready for tenants". Day 1 isolates and provisions a host for a tenant. Day 2 keeps the fleet healthy and returns released hosts to the pool.
Day 0: Discovery, Validation, and Ingestion: The provider registers each host as an Expected Machine with its BMC MAC address, chassis serial, and factory BMC credentials. From there NICo takes over: it discovers the host over Redfish, links each DPU to its host, validates the machine against its SKU, runs burn-in and fabric tests, brings firmware to the site baseline, installs the DPU OS and HBN, attests the host with measured boot and TPM checks when attestation is enabled, and allocates its IP addresses. Component inventory is collected automatically, so no manual component entry is required.
Day 1: Isolation, Lockdown, and Provisioning: Before a tenant receives a host, NICo establishes isolation on every network plane, locks down UEFI and the BMC, and coordinates the iPXE boot of the tenant's chosen OS image. The provider grants the tenant an allocation of hosts, and the tenant creates instances from it through the REST API.
Day 2: Operations, Health, and Tenant Transitions: NICo monitors health continuously, schedules firmware upgrades on unoccupied hosts, upgrades itself in place, and sanitizes every released host before it returns to the available pool, re-attesting it when attestation is enabled.
Refer to Day 0, Day 1, and Day 2 Lifecycle for the detail behind each step.
Each flow below is an end-to-end procedure you can run against a site. The
provider-side flows use nicocli or the REST API, and the REST API Reference
documents the matching API call sequences.
| Flow | Phase | Guide |
|---|---|---|
| Register a site and ingest its first hosts | Day 0 | Ingesting Hosts and Ingesting Hosts (REST API) |
| Validate machines against their SKU and run burn-in | Day 0 | SKU Validation and Machine Validation |
| Create a tenant, allocate hosts, and provision instances | Day 1 | Tenant Management |
| Isolate tenant networks across Ethernet, InfiniBand, and NVLink | Day 1 | Network Isolation |
| Upgrade host, DPU, and rack component firmware | Day 2 | Firmware Updates, with host, DPU, and rack component workflows |
| Release an instance and verify sanitization | Day 2 | Tenant Lifecycle Cleanup |
| Monitor fleet health and respond to alerts | Day 2 | Monitoring Health |
| Run rack-level power and maintenance operations with NICo Flow | Day 2 | NICo Flow Operations |
| Upgrade NICo itself | Day 2 | Upgrading NICo |
NICo sits below Kubernetes and platform layers. It exposes REST and gRPC APIs that higher-level systems consume directly, and it does not dictate how scheduling, tenancy policy, or workloads are managed above it.
- Kubernetes, SLURM, and workload schedulers run above NICo. NICo provisions hosts and hands them off at boot. Assembling them into clusters, scheduling workloads, and patching the OS belong to the platform layer.
- BMaaS, VMaaS, and ISV control planes consume the NICo REST and gRPC APIs to allocate and provision bare metal, and they own tenancy policy.
- UFM, NMX-C, and the switch fabric stay under the network team. NICo calls the UFM and NMX-C APIs for InfiniBand and NVLink partitions and programs isolation on the DPU, but it does not configure switches or install UFM, NMX-C, or Cumulus Linux.
- DPF and RMS are NVIDIA components that NICo integrates with. The DOCA Platform Framework provisions DPUs for NICo, and the Rack Management Service carries out rack-level hardware operations that Flow sequences.
- PostgreSQL, Vault, Temporal, cert-manager, and the telemetry stack are off-the-shelf dependencies. NICo uses them but does not operate them.
Refer to Scope and Boundaries for the full division of responsibility.
The Site Controller is the on-site control plane: Rust services deployed on a Kubernetes cluster of at least three nodes, all talking to each other over mutual TLS (mTLS) and gRPC, plus the Go Site Agent that links the site to NICo REST. NICo REST is a Go service layer that exposes those capabilities as a REST API. It can run next to the Site Controller or centrally in the cloud, where many sites connect to it through their Site Agents.
| Service | Role |
|---|---|
| API Service (NICo Core) | Single source of truth and the only writer to the Core database. Runs the state machines for hosts, network segments, and InfiniBand and NVLink partitions. |
| DHCP, PXE, and DNS | Stateless front doors for the physical network. They forward every decision to NICo Core, which owns IP address management and boot artifacts. |
| Hardware Health | Scrapes host and DPU BMCs over Redfish and exports Prometheus metrics and per-sensor telemetry. |
| SSH Console | Holds persistent BMC console sessions, streams them to Loki, and gives tenants and operators live serial access. |
| Site Agent and NICo REST | Connect the on-site controller to the REST API over Temporal, so operators and ISVs integrate with HTTP and JWT. |
| NICo Flow | Rack inventory and durable orchestration of power, firmware, ingestion, and bring-up operations, grouped by rack. Persists its tasks in its own PostgreSQL database and runs them through Temporal. |
| Scout and DPU Agent | Run on the managed hosts. Scout collects in-band inventory and runs validation during discovery. The DPU agent applies network state, serves instance metadata, and self-updates. |
Refer to Architecture: Overview and Components for the full component list and to the state handling design for how the state machines work.
There are two ways to run NICo. Use the local development stack to work on the code against mock hardware. Use the production installer to deploy a complete site onto a bare-metal Kubernetes cluster.
The fastest way to run the whole stack locally is DevSpace on a kind cluster. It compiles the Core and REST images from this clone, deploys the Helm charts, and uses Machine-a-Tron to supply mock hosts, so you can exercise the REST API and the Core gRPC API end to end without hardware.
# 1. Install Docker, kind, kubectl, helm, and devspace. DevSpace builds every
# image inside Docker, so no Rust or Go toolchain is needed for this path.
# 2. Create a local cluster. The bootstrap script works on your current
# Kubernetes context, so this must exist first.
kind create cluster --name nico
# 3. Bootstrap the cluster-side prerequisites (cert-manager, PostgreSQL, Vault,
# Temporal, and Keycloak) on that context. Safe to re-run.
dev/deployment/devspace/bootstrap-prereqs.sh
# 4. Build and deploy NICo Core, NICo REST, and the mock hosts.
devspace deployFor a plain build and test loop without a cluster, install rustup with the
Rust version pinned in rust-toolchain.toml, the Go version in
rest-api/go.mod, and cargo install cargo-make, then:
cargo build # all Rust workspace crates
cargo make correctly-execute-tests # build prerequisites, then run the Rust tests
make rest-build # Go REST API binaries
make rest-test # Go unit tests with PostgreSQL and mock gRPC servers
cargo make pre-commit-verify-workspace # what CI runs before mergeOn a fresh Ubuntu VM, dev/deployment/devspace/setup-devspace-on-host.sh
performs all four steps, including the kind cluster.
Refer to Local Development with DevSpace for profiles, port-forwarding, and teardown, to the Development page for host package prerequisites, to the Build Guide for ARM64 and release builds, and to AGENTS.md for the full build, test, and lint command reference.
helm-prereqs/setup.sh deploys the full NICo stack onto a bare-metal
Kubernetes cluster in layers. The first three always install. DPF and RMS
install by default and can be skipped. Flow installs whenever REST does.
| Layer | What it installs | Helm release |
|---|---|---|
| Common services | MetalLB, cert-manager, Vault, External Secrets Operator, and PostgreSQL | via helmfile in helm-prereqs/ |
| NICo Core | The Site Controller services from this repo's helm/ chart |
nico in nico-system |
| NICo REST | The REST API, Temporal, Keycloak, and Site Agent | nico-rest and nico-rest-site-agent in nico-rest |
| NICo Flow | Rack-level task and automation service | flow in flow |
DPF (--skip-dpf to opt out) |
The DOCA Platform Framework stack that provisions DPUs | dpf-operator in dpf-operator-system |
RMS (--skip-rms to opt out) |
The Rack Management Service | rack in rack-manager |
Refer to What gets deployed for the complete list with chart versions.
- A running Kubernetes cluster of at least three nodes with
KUBECONFIGset helm,helmfile,kubectl, andjqinstalled- Images pushed to your container registry
# 1. Build and push amd64/arm64 container images from this clone.
# Refer to docs/manuals/building_nico_containers.md for the build-host prerequisites.
# The values below are examples: replace every one of them with your own.
export IMAGE_REGISTRY=my-registry.example.com/infra-controller
export IMAGE_TAG=v2.2.0
make images IMAGE_REGISTRY="${IMAGE_REGISTRY}" IMAGE_TAG="${IMAGE_TAG}" # NICo Core and REST service images
# 2. Set environment variables, reusing the registry and tag from step 1.
export KUBECONFIG=/path/to/kubeconfig
export NICO_IMAGE_REGISTRY="${IMAGE_REGISTRY}"
export NICO_CORE_IMAGE_TAG="${IMAGE_TAG}"
export NICO_REST_IMAGE_TAG="${IMAGE_TAG}"
# For an authenticated registry, read the pull secret from a prompt so it
# never appears in shell history:
# read -r -s -p 'Registry pull secret: ' REGISTRY_PULL_SECRET && export REGISTRY_PULL_SECRET
# DPF (DOCA Platform Framework) DPU provisioning installs by default.
# Set these two variables, or pass --skip-dpf to opt out:
export NICO_DPF_DPU_INTERFACE=ens1f0 # controller NIC facing the DPUs
export NICO_DPF_DPU_CLUSTER_VIP=192.0.2.10 # free routable IP for the DPU cluster control plane
# RMS (Rack Management Service) installs by default and requires an image
# tag. Its default image is entitlement-gated on NGC, so also supply an NGC
# API key (it defaults to REGISTRY_PULL_SECRET), point NICO_RMS_IMAGE_REPO
# at your mirror, or pass --skip-rms to opt out:
export NICO_RMS_IMAGE_TAG=v0.8.0
read -r -s -p 'NGC API key for RMS: ' NICO_RMS_NGC_API_KEY && export NICO_RMS_NGC_API_KEY
# 3. Customize site-specific values
# Edit helm-prereqs/values/nico-core.yaml:
# nico-api.hostname - your site's external API hostname
# nico-api.siteConfig - network pools, VLAN ranges, IB config, MetalLB VIPs
# Edit helm-prereqs/values/metallb-config.yaml:
# IPAddressPool, BGPPeer - your site's VIP ranges and TOR switch config
# Edit helm-prereqs/values.yaml:
# siteName - short site identifier
# 4. Run setup. It installs common services, NICo Core, and NICo REST in order.
cd helm-prereqs
./setup.sh # interactive: prompts before deploying Core and REST
./setup.sh -y # non-interactive: deploys everything including DPF (CI/CD)
./setup.sh -y --skip-dpf # non-interactive: skip DPF (no DPUs, or still on iPXE)To tear everything down:
cd helm-prereqs
./clean.shRefer to the Quick Start Guide for the
step-by-step walkthrough, including the siteConfig block and first host
discovery, and to helm-prereqs/README.md for the
full reference: PKI architecture, PostgreSQL setup, phase-by-phase description,
every environment variable, DPF options, secrets, health checks, and
troubleshooting.
Runnable inputs and reference configurations that ship with the repository:
| Example | What it shows |
|---|---|
| Go Simple SDK examples | Small programs against the REST API for machines, expected machines, instances, VPCs, and IP blocks, built on the Simple SDK |
| Site bootstrap manifest | Input for nicocli site bootstrap, which initializes the calling organization and creates the instance types, allocations, VPCs, and VPC prefixes for a Site that already exists |
| NICo Flow inputs | A GB200 NVL72 rack definition and operation rules for the Flow CLI |
| Helm values | Minimal and full values files for installing the NICo Core chart directly |
| Path | What lives there |
|---|---|
crates/ |
The Rust workspace: NICo Core (crates/api), DHCP, PXE, DNS, hardware health, the DPU agent, Scout, the admin CLI, and shared libraries |
rest-api/ |
The Go REST API, Site Agent, NICo Flow, nicocli, and the OpenAPI specification and generated SDK |
helm/ and helm-prereqs/ |
The NICo Core Helm chart and the setup.sh reference installer for a complete site |
docs/ |
Source for the published documentation, including architecture, provisioning, and operations guides |
dev/ |
Local development tooling, including the DevSpace deployment with mock hardware |
bluefield/ and pxe/ |
DPU image components and PXE boot artifact generation |
deploy/ |
Kubernetes manifests and Kustomize overlays |
NICo publishes a three-month rolling roadmap as a pinned GitHub issue, maintained alongside the monthly release cadence. It is directional rather than a commitment, and each item links to its tracking issue.
- Public roadmap
- Milestones for the current implementation status of each release
- Release Policy for how the roadmap and the release cycle fit together
- Chat: join the NICo community Slack
- Questions and ideas: GitHub Discussions
- Bugs and feature requests: GitHub Issues
- Releases: Release Notes for what changed, and the Release Policy for the monthly branch, version, and support cadence
- Security: report vulnerabilities as described in SECURITY.md
Contributions are welcome. Refer to the contribution guide for the development environment, the DCO sign-off process, and the engineering guidelines, and to the code of conduct for contributor expectations.
NICo is licensed under the Apache License 2.0.
NICo is under active development. The software is provided "as is" without warranties of any kind. Features, APIs, and configurations may change in future releases. For production deployments, please test thoroughly in non-critical environments first.