Skip to content

feat: fresh Ubuntu testing#2312

Open
paul-nechifor wants to merge 1 commit into
mainfrom
paul/feat/clean-testing
Open

feat: fresh Ubuntu testing#2312
paul-nechifor wants to merge 1 commit into
mainfrom
paul/feat/clean-testing

Conversation

@paul-nechifor
Copy link
Copy Markdown
Contributor

Problem

  • We usually test in a provisioned environment. This means we don't know if the install and test instructions actually work for new users.
  • Also, we need to do some manual tests before a release.

Closes DIM-968

Solution

  • Add a script which:
    • downloads Ubuntu Desktop
    • installs it in VirtualBox
    • runs our setup instructions and testing instructions

How to Test

./misc/fresh-ubuntu-tests/vmtest.sh build
./misc/fresh-ubuntu-tests/vmtest.sh run

Contributor License Agreement

  • I have read and approved the CLA.

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 30, 2026

Greptile Summary

Adds a local VirtualBox-based test harness (misc/fresh-ubuntu-tests/) that downloads a fresh Ubuntu 24.04 Desktop ISO, performs an unattended install, and replays the documented dimos install + test flow inside the VM — catching gaps between docs and a clean machine that CI's cached images cannot detect. Also includes minor cleanup: doclinks.py now prints per-file output only when there is something to report, and the DDS docs drop a redundant uv pip install step.

  • vmtest.sh manages the VirtualBox lifecycle: build creates and snapshots the golden VM, run clones it and executes the suite, clean removes stale clones.
  • suite/ holds setup.sh (apt deps + uv + git clone), run.sh (runs each test script, collects per-test logs, reports PASS/FAIL), and three test scripts for mypy, pytest, and CycloneDDS.

Confidence Score: 4/5

Safe to merge as a local tooling addition; no production code is affected, but vmtest.sh has a logic flaw in ISO handling that will cause confusing build failures for anyone whose download was interrupted.

The ensure_iso() function returns early whenever the ISO file exists on disk, making the wget --continue flag dead code. A partially-downloaded ISO is silently treated as complete, and the only symptom is a cryptic VirtualBox installer failure ~20 minutes into the build. All other files — the suite scripts, doclinks.py cleanup, and docs — look correct.

misc/fresh-ubuntu-tests/vmtest.sh — specifically the ensure_iso() function and the shared SSH port handling.

Important Files Changed

Filename Overview
misc/fresh-ubuntu-tests/vmtest.sh Main VirtualBox orchestration script; has a real logic bug where a partial ISO download is silently accepted as valid, plus missing SHA-256 verification and a shared NAT port that can conflict if the golden VM is still running.
misc/fresh-ubuntu-tests/suite/run.sh Test runner executed inside the VM; correctly uses pipefail + PIPESTATUS to capture per-test exit codes while continuing on failure. No issues found.
misc/fresh-ubuntu-tests/suite/setup.sh Installs system deps, uv, and clones the repo from main; straightforward replay of the documented install flow. No issues found.
misc/fresh-ubuntu-tests/suite/tests/03-cyclonedds.sh Installs CycloneDDS, creates the compatibility symlink tree, wipes and re-creates the venv with the dds extra. Matches the updated docs/usage/transports/dds.md instructions.
dimos/utils/docs/doclinks.py Moves the "Processing …" header to print only when changes or errors exist, and removes the noisy "No changes needed" line; clean improvement with no logic regressions.
docs/usage/transports/dds.md Removes the redundant uv pip install -e '.[dds]' step and updates the sync command to --all-extras --all-groups, consistent with the new test script.
docs/development/testing.md Adds documentation for the new VM-based test harness with usage instructions. No issues.
.gitignore Adds /misc/fresh-ubuntu-tests/cache to gitignore to exclude downloaded ISOs, VMs, SSH keys, and logs. Correct and necessary.

Sequence Diagram

sequenceDiagram
    participant User
    participant vmtest.sh
    participant VBoxManage
    participant GoldenVM
    participant CloneVM
    participant suite/run.sh

    Note over User,suite/run.sh: ./vmtest.sh build (once)
    User->>vmtest.sh: build
    vmtest.sh->>vmtest.sh: ensure_iso() — wget ISO
    vmtest.sh->>VBoxManage: createvm + unattended install
    VBoxManage->>GoldenVM: start headless, install Ubuntu
    GoldenVM-->>vmtest.sh: SSH ready (/etc/dimos-vmtest-ready)
    vmtest.sh->>VBoxManage: power off + snapshot "golden"

    Note over User,suite/run.sh: ./vmtest.sh run (each time)
    User->>vmtest.sh: run
    vmtest.sh->>VBoxManage: clonevm golden --options link → CloneVM
    VBoxManage->>CloneVM: startvm headless
    CloneVM-->>vmtest.sh: SSH ready
    vmtest.sh->>CloneVM: scp suite/
    vmtest.sh->>CloneVM: ssh bash suite/run.sh
    CloneVM->>suite/run.sh: setup.sh (apt-get, uv, git clone)
    suite/run.sh->>suite/run.sh: 01-mypy.sh
    suite/run.sh->>suite/run.sh: 02-pytest.sh
    suite/run.sh->>suite/run.sh: 03-cyclonedds.sh
    suite/run.sh-->>vmtest.sh: PASS / FAIL + exit code
    vmtest.sh->>CloneVM: scp logs back
    vmtest.sh->>VBoxManage: remove CloneVM (trap EXIT)
Loading

Reviews (1): Last reviewed commit: "feat: fresh ubuntu testing" | Re-trigger Greptile

Comment on lines +105 to +110
if [[ -f "$ISO_PATH" ]]; then
return
fi

wget --tries=3 --continue -O "$ISO_PATH" "https://releases.ubuntu.com/$RELEASE/$fname" \
|| die "download failed"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Partial download silently accepted as a valid ISO. The if [[ -f "$ISO_PATH" ]]; then return; fi guard fires as soon as the file exists — even if a previous wget was interrupted mid-download. The --continue flag on the wget call below is unreachable in this case, so the partial file is passed to VirtualBox as-is. The unattended install will then fail with a cryptic installer error rather than a clear "ISO is corrupt" message. The fix is to skip the early return and let wget --continue resume (or verify the file is complete after creation).

Suggested change
if [[ -f "$ISO_PATH" ]]; then
return
fi
wget --tries=3 --continue -O "$ISO_PATH" "https://releases.ubuntu.com/$RELEASE/$fname" \
|| die "download failed"
wget --tries=3 --continue -O "$ISO_PATH" "https://releases.ubuntu.com/$RELEASE/$fname" \
|| die "download failed"

Comment on lines +29 to +30
readonly RELEASE="24.04"
readonly VM="dimos-vmtest-base"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 The minor point-release version (4) in the ISO filename is hardcoded separately from the RELEASE constant. When Ubuntu publishes 24.04.5, this script will silently keep downloading the older .4 image (or get a 404 if Canonical eventually removes it). Deriving the full filename from a single versioned constant keeps them in sync and makes future bumps a one-liner.

Suggested change
readonly RELEASE="24.04"
readonly VM="dimos-vmtest-base"
readonly RELEASE="24.04"
readonly RELEASE_POINT="24.04.4" # bump when Ubuntu publishes a new point release
readonly VM="dimos-vmtest-base"

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +99 to +111
ensure_iso() {
need wget
mkdir -p "$ISO_DIR"
local fname="ubuntu-24.04.4-desktop-amd64.iso"
ISO_PATH="$ISO_DIR/$fname"

if [[ -f "$ISO_PATH" ]]; then
return
fi

wget --tries=3 --continue -O "$ISO_PATH" "https://releases.ubuntu.com/$RELEASE/$fname" \
|| die "download failed"
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 No SHA-256 integrity check after download. Ubuntu publishes SHA256SUMS and SHA256SUMS.gpg alongside every ISO at https://releases.ubuntu.com/24.04/. Without verifying the checksum, a corrupted download (e.g., from a flaky connection, CDN error, or partial wget resume misalignment) passes silently into VirtualBox. Adding a sha256sum -c step after wget completes — using the checksum fetched from Ubuntu — would catch bad downloads before the ~15-30 min install begins.

readonly MEM=12288 # MiB
readonly CPUS=8
readonly DISK=40960 # MiB (dynamic; only grows as used)
readonly SSH_PORT=2222
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Hardcoded SSH_PORT shared between golden VM and all clones. The golden VM's NAT rule binds 127.0.0.1:2222 → :22 at build time. Every linked clone inherits the same rule. If the golden VM is still running (e.g., left up after a previous build) when run starts its clone, both VMs compete for port 2222; VirtualBox doesn't error out but the SSH connection may reach the wrong VM or fail intermittently. A simple guard (vm_state "$VM" != "running") at the start of cmd_run — or stopping the golden VM if it is running — would prevent this.

@codecov
Copy link
Copy Markdown

codecov Bot commented May 30, 2026

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/utils/docs/doclinks.py 0.00% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant