Skip to content

test(e2e): log runner VM consoles when no runner registers - #426

Closed
cbartz wants to merge 1 commit into
mainfrom
test/e2e-capture-runner-console
Closed

cbartz wants to merge 1 commit into
mainfrom
test/e2e-capture-runner-console

Conversation

@cbartz

@cbartz cbartz commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

What this PR does

While the GARM E2E waits for a runner to register, it now snapshots the serial console of every running runner VM (Nova os-getConsoleOutput, at most once a minute per VM). If no runner registers, it logs the last 300 lines of each VM's latest snapshot, redacted, next to the existing GARM debug dump.

Why we need it

The nightly edge-to-candidate promotion has failed since 2026-10-08: runner VMs boot but never contact GARM. Only the VM console can show whether the callback tunnel came up or where cloud-init stopped, and GARM's bootstrap-timeout reaper deletes the VM before the test gives up. So the console has to be read while the VM is still up.

Test plan

  • tox -e charms-tests-unit: new model-free tests cover the fetch, throttling, error tolerance and redaction, plus the wait handing every poll to the recorder.
  • A manual garm_e2e.yaml dispatch on this branch, to check the console shows up in a real failing run.

Review focus

  • Redaction: the log is public. Output goes through the existing credential sentinels, then JWT, Bearer, --token and token=/token: patterns. GARM's install script only traces its tokens with boot debug on, so the patterns are defence in depth.
  • Error handling: OpenStack errors are only logged, so diagnostics can never change a test's outcome.
  • No new dependencies: Keystone and Nova are called with plain requests, using the OS_* credentials the suite already has.

Checklist

  • Changes comply with the project's coding standards and guidelines (see CONTRIBUTING.md and STYLE.md)
  • CONTRIBUTING.md has been updated upon changes to the contribution/development process (e.g. changes to the way tests are run)
  • Technical author has been assigned to review the PR in case of documentation changes (usually *.md files)
  • I updated docs/changelog.md with user-relevant changes
  • I used AI to assist with preparing this PR
  • I added or updated tests as needed (unit and integration)
  • If integration test modules are used: I updated the workflow configuration
  • If this PR involves a Grafana dashboard: I added a screenshot of the dashboard
  • If this PR involves Terraform: terraform fmt passes and tflint reports no errors
  • If this PR involves Rockcraft: I updated the version
  • If this PR adds/removes a charm, or changes a charm's base class, conventions, tooling, or repo structure: I updated the relevant AGENTS.md
  • If this PR changes .copilot-collections.yaml or .github/instructions/: I re-checked whether the AGENTS.md "12-factor divergences" guidance still matches the upstream copilot-collections guidance

@cbartz

cbartz commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator Author

Closing: the console capture served its purpose — it showed the runner VMs' SSH tunnel to the CI host timing out, which #427 fixes by replacing the tunnel with a tenant relay. Not needed as a permanent diagnostic for now.

AI-generated message.

@cbartz cbartz closed this Oct 14, 2026
@cbartz
cbartz deleted the test/e2e-capture-runner-console branch October 14, 2026 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant