Issues

• [ALERT] Access to NVIDIA's Self-Hosted Runners is Expiring

  nvidia-runners
  NVIDIA-NeMo/Evaluator#1154 · Filter by author nvidia-runner-mgmt opened on Aug 30, 2026

• SkillsEnvironment silently degrades to exact string match when the NeMo Skills evaluator cannot be constructed or raises

  community-request
  NVIDIA-NeMo/Evaluator#1141 · Filter by author yingguo-trt opened on Aug 11, 2026
  1

• vlmevalkit:// MCQ scoring does not preserve VLMEvalKit judge-based answer extraction; configured JudgeMetric is not wired into environment verification

  community-requestwaiting-on-maintainers
  NVIDIA-NeMo/Evaluator#1140 · Filter by author yingguo-trt opened on Aug 11, 2026
  7

• TriviaQA loading fails: legacy trivia_qa id rejected by HF Hub namespace/name validation

  bugcommunity-requestwaiting-on-maintainers
  NVIDIA-NeMo/Evaluator#1111 · Filter by author cris96spa opened on Jul 10, 2026

• XSTest loading fails

  bugcommunity-requestwaiting-on-maintainers
  NVIDIA-NeMo/Evaluator#1110 · Filter by author cris96spa opened on Jul 10, 2026

• HealthBench fallback URL 404

  bugcommunity-requestwaiting-on-maintainers
  NVIDIA-NeMo/Evaluator#1109 · Filter by author cris96spa opened on Jul 10, 2026

• lm-evaluation-harness container: shared aiohttp ClientSession with ClientTimeout(total=...) aborts long benchmark runs (e.g. MMLU-Pro 12k prompts)

  community-requestwaiting-on-maintainers
  NVIDIA-NeMo/Evaluator#974 · Filter by author shauryr opened on May 6, 2026
  1

  

• Evaluation support for multimodal models in Megatron Bridge

  enhancement
  Feature

  NVIDIA-NeMo/Evaluator#279 · Filter by author athitten opened on Oct 6, 2025 · 26.04
  12

  

• [Feature Request] Support chat format in the llm generate APi

  support
  NVIDIA-NeMo/Evaluator#49 · Filter by author snowmanwwg opened on Aug 4, 2025
  4