feat: redesign FastSAC console rich panel (#15)

[wlgys8]

Target branch

Branch on Motphys/MotrixLab targeting main (same-repo PR, no fork involved).

Summary

Redesigns the FastSAC Rich training panel (#15):

• Information hierarchy: one summary row of five cards (Run progress with a
  bar, Episode stats, Throughput, Timing, System health), followed by
  Training / Rewards / Environment metric cards with terminal-width-adaptive
  compact grids (2–4 columns).
• Detail view: 1/2 keys switch between Overview and a Timing view that
  renders per-process timing trees with MEAN/SHARE columns, plus timing detail
  and diagnostics. Keyboard input uses a cbreak _InputLive subclass and is
  POSIX-only; other platforms fall back to a plain Live.
• System health: CPU load, GPU utilization, host RAM and VRAM. GPU metrics
  are sampled in-process via NVML (nvidia-ml-py) instead of spawning
  nvidia-smi per refresh (steady-state cost drops from ~2 spawns × tens of ms
  to ~5 µs); host memory sampling supports Linux /proc/meminfo and Windows
  GlobalMemoryStatusEx.
• Checkpoint feedback: saved checkpoint paths render inside the panel
  instead of interrupting the live display.
• No-wrap metric cells: values exceeding the 9-char value cell fall back to
  two-digit scientific notation, and label width is derived from the exact Rich
  column arithmetic, so metric rows stay on one line from narrow to wide
  terminals.
• The plain-text fallback (non-TTY / no Rich) is unchanged apart from the async
  learner's learn_ms semantics (now the per-update total of drain + waits +
  update stages; TensorBoard scalar renamed perf/learn_ms_per_update →
  perf/learn_ms_total).

Issue

Fixes #15

Validation

• uv run pytest -q (883 passed)
• prek run --all-files
• CI checks pass on the target branch
• Documentation updated (if applicable) — not applicable
• Third-party attribution updated (if applicable) — nvidia-ml-py added to
  the runtime dependency list in THIRD_PARTY_NOTICES.md
• No credentials, secrets, private URLs, or sensitive robot data are
  included in the commits, logs, or screenshots.

Additional manual validation: panel rendering smoke-tested at 80–300 terminal
columns in both views with no markup leakage; Windows degradation verified by
importing with termios/tty hidden; NVML sampling benchmarked on a live GPU.

Compatibility and migration

• New runtime dependency nvidia-ml-py>=13.610.43 on motrix_rl (pure Python
  ctypes bindings, locked in uv.lock). GPU panel fields show n/a when no
  NVIDIA driver is present.
• motrix_rl.console no longer imports termios/tty unconditionally, so it
  is importable on Windows; keyboard view switching remains POSIX-only.
• TrainingPanelStats gained optional fields (gpu_utilization_percent,
  memory_usage, gpu_memory_usage, checkpoint_path); existing consumers
  are unaffected.
• Internal: run_learner_process gained an env_name parameter; panel titles
  now include the environment name.

Safety and release impact

• Performance: removes two nvidia-smi subprocess spawns per panel refresh
  from the learner step loop (previously ~16 ms each on reference hardware,
  worse under load); NVML sampling is microseconds.
• No asset, license, or hardware changes. Recommend squash merge to keep a
  single commit on main.

[Copilot]

🟡 Changes recommended

There are correctness issues in the new console interaction and async learn_ms_total computation that can misreport behavior and/or leave terminal state inconsistent on exceptions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR redesigns the FastSAC Rich live training panel in motrix_rl.console to improve information hierarchy, add an overview vs. timing-detail view, and surface “system health” (CPU/GPU + RAM/VRAM) with lower refresh overhead by sampling NVIDIA GPU metrics via NVML.

Changes:

• Reworked Rich panel renderer (summary cards + compact metric grids) and added a timing detail view toggled via 1/2 keys on POSIX TTYs.
• Added host memory + GPU utilization/VRAM samplers and integrated them into sync/async FastSAC loops; checkpoint saves now surface in-panel.
• Introduced runtime dependency nvidia-ml-py and expanded tests/attribution accordingly.

File summaries

File	Description
uv.lock	Locks nvidia-ml-py and records its distribution artifacts.
THIRD_PARTY_NOTICES.md	Adds nvidia-ml-py to runtime dependency attribution.
motrix_rl/pyproject.toml	Declares new runtime dependency nvidia-ml-py>=13.610.43.
motrix_rl/src/motrix_rl/system_metrics.py	Adds MemoryUsage + Linux/Windows RAM sampling and NVML-based GPU samplers.
motrix_rl/src/motrix_rl/console.py	Implements new Rich panel layout, compact metric formatting, and POSIX-only key input handling.
motrix_rl/src/motrix_rl/fastsac/sync/train.py	Wires system samplers + checkpoint path into sync training panel stats/title.
motrix_rl/src/motrix_rl/fastsac/async_impl/worker.py	Updates async learner timing semantics/tree and wires system samplers + checkpoint path + renamed TB scalar.
motrix_rl/src/motrix_rl/fastsac/async_impl/train.py	Passes env_name into learner process for panel titles.
motrix_rl/tests/test_system_metrics.py	Adds unit tests for NVML aggregation/error handling and Linux/Windows memory sampling.
motrix_rl/tests/test_console.py	Adds rendering/layout/timing-view tests and POSIX-TTY degradation coverage.

Review details

• Files reviewed: 9/10 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟢 Approval recommended

The redesign is well-covered by targeted tests and the functional changes appear correct, with only minor maintainability follow-ups noted in review comments.

Review details

Suppressed comments (1)

motrix_rl/src/motrix_rl/console.py:505

• The nested si() helper inside render_training_panel() is unused. Keeping unused helpers in the hot rendering path makes the function harder to read and suggests incomplete refactors.

    def si(number: float) -> str:
        for unit in ("", "k", "M"):
            if abs(number) < 1000:
                return f"{number:.0f}{unit}" if not unit else f"{number:.1f}{unit}"
            number /= 1000.0
        return f"{number:.1f}G"

• Files reviewed: 10/11 changed files
• Comments generated: 2
• Review effort level: Lite

[wlgys8]

All review feedback is addressed in 8acd012 (branch squashed back to a single commit):

Round 1

• _InputLive.start(): the whole setup (fd acquisition + termios changes + super().start()) now sits in a single try/except that calls _restore_stdin(), so a mid-setup failure restores the saved TTY state and closes an opened /dev/tty fd.
• async learn_ms: each learner-loop pass is timed end to end, and _window_learn_ms_total() divides the window's summed stage wall-clock by its update count (0 when no updates) — a true per-update total instead of a sum of per-stage means with differing sample counts.
• keyboard hint: rendered only when _POSIX_TTY, so platforms falling back to a plain Live no longer advertise 1/2 switching.

Round 2

• TrainingPanelStats.memory_usage / gpu_memory_usage are typed MemoryUsage | None (with _format_memory and the nested memory_style helper matching), so the .used_bytes/.total_bytes contract is static-checked.
• Removed the unreferenced _prototype_timing_summary and the unused nested si() in render_training_panel().

Validation: uv run pytest -q → 886 passed; prek run clean.

[wlgys8]