Repository navigation
Make compute-image builds follow the selected queue system - #79
Merged
Merged
Conversation
The confluent and xCAT compute-image builds staged SLURM unconditionally: munge.key installed with owner munge:munge, slurm.conf synced, and munge/slurmd enabled in every image. With PBS selected the munge user never exists on the head node, so the confluent image build aborted with "install: invalid user 'munge'", and the xCAT synclists referenced files PBS installs do not have. Gate the SLURM staging on QueueSystem::Kind and add the PBS equivalent: install openpbs-execution-ohpc in the image, point pbs.conf at the head node, and enable the pbs service (the execution package's default pbs.conf starts only the MOM daemon). Clusters without a queue system still get the base OHPC tooling. - confluent: new buildNodeImageQueueSystemCommands() emits the per-queue image block; package lists take the queue kind and the SLURM output is byte-identical to before. - xCAT: configureSLURM()/new configurePBS() are selected by queue kind, generateSynclistsFile() now writes the accumulated per-queue synclists instead of a hardcoded SLURM list, and the munge permission fixes no longer dereference an empty queue-system optional. Fixes #63
The PBS role configured server attributes but never normalized the OHPC placeholder PBS_SERVER in /etc/pbs.conf and never created the compute nodes on the server, so MOMs from the (now queue-aware) images had nothing to register against. Set PBS_SERVER to the head node before starting the service and create each cluster node with qmgr.
…tu base Codex review findings on the queue-aware image work: - PBS node registration ran in the queue-system role, before makehosts / confluent2hosts publish compute host records; qmgr resolves names at creation time, so on fresh installs every create failed silently. Move registration into both provisioners' add-nodes flows, after host generation, as an idempotent list-or-create composite that fails loudly on real errors. - The Ubuntu 24.04 xCAT base package set hardcoded ohpc-slurm-client, so PBS and no-queue images still shipped SLURM. The scheduler client now comes only from configureSLURM()/configurePBS().
confluent2hosts drops the head node's management /etc/hosts entry (the gap buildHeadnodeHostsRepairScript exists to close), and PBS_SERVER uses that short name, so registering nodes inside addNodes() could not reach the server on fresh installs. Run the registration after the repair, right before osdeploy initialize -l.
Codex review: the OHPC Debian OpenPBS packages ship no default /etc/pbs.conf and run no postinstall initializer, so sed-editing the file aborted Ubuntu image builds under set -e and left the head-node role unconfigured. Write the canonical pbs.conf (server flavor on the head node, MOM flavor in compute images) on all platforms; the pbs service's first start runs pbs_habitat to create PBS_HOME.
Codex review: shouldReuseExistingImage() keyed only on the osimage name, so a rerun that switched SLURM to PBS reused the stale SLURM image while PBS registration still ran, booting nodes without a MOM. Stamp a compute.queue marker when an image is built and force a rebuild when the marker is missing or names a different queue.
Codex review round on the queue-aware images: - On EL images the execution RPM's %post already ran pbs_habitat against the package placeholder, so only rewriting pbs.conf left mom_priv's clienthost pointing at the placeholder. Rerun pbs_habitat after the real pbs.conf is in place and write the MOM config outright on both provisioner paths. - Scope the xCAT queue marker per osimage (compute.queue.<osimage>) so cached images for different queues do not validate each other, and stamp it only after genimage succeeds so a failed rebuild cannot pass the reuse check on the next run.
Codex review: on a same-queue forced rebuild the previous marker survived until the post-genimage stamp, so a rebuild failing midway left a matching marker and the next run reused the partial root image. Remove the marker as soon as the rebuild path is entered.
Codex review: the synclist path is shared between cached osimages while the queue markers are per image, so reusing image A after building image B for another queue repacked A with B's scheduler files (e.g. a PBS image receiving slurm.conf and the munge key). Build the synclist content directly from the selected queue and regenerate it before every repack, on the reuse path included; the m_stateless synclists accumulator is gone.
Codex review: OpenPBS caches the MOM address when a node is created, so the list-or-create registration stranded nodes whose management IP changed across reinstalls while keeping their hostname. Delete and recreate each node instead; the delete may fail quietly (node absent), only the create aborts the run.
…ches Codex review: unconditionally deleting and recreating nodes discarded administrator-set attributes and aborted the run on busy nodes (delete refused, create then failing on the existing entry). Go back to creating only missing nodes and restart the pbs service once after registration: the restart re-resolves cached MOM addresses against the freshly published host records without touching node definitions, covering the changed-IP reinstall case non-destructively.
Codex review: a rerun with PBS still selected but a renamed head node reused the cached image whose baked-in pbs.conf and mom_priv/config still name the old server. Append the head-node hostname to the PBS marker so such images are rebuilt; SLURM needs no fingerprint since slurm.conf is re-synced on every repack.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The confluent and xCAT compute-image builds staged SLURM unconditionally (munge.key owned by a nonexistent munge user, slurm.conf, munge/slurmd enabled), so PBS installs aborted with
install: invalid user 'munge'. Beyond the crash, PBS had no compute-side setup at all.What this does
QueueSystem::Kindin both provisioner paths; SLURM output is byte-identical to before.openpbs-execution-ohpcin the image, canonicalpbs.confwritten outright (the OHPC Debian packages ship none),pbs_habitatrerun + MOM config written so RPM images don't keep the package placeholder server, and thepbsservice enabled. No-queue clusters keep base OHPC tooling.pbs.conf, and compute nodes registered with the server aftermakehosts/confluent2hostspublish host records (create-if-missing, then onepbsrestart to re-resolve cached MOM addresses non-destructively).genimagesucceeds and dropped when entering a rebuild, and the shared synclist regenerated from the queue selection before every repack.Review/validation: 10 iterative Codex review rounds; every finding in the changed code addressed (registration timing, Ubuntu package leakage, DEB config assumptions, cache-staleness cases). Container preflight green throughout, finally 1265/1265 assertions; new unit tests cover the per-queue package lists, image blocks, registration command, and marker tokens.
Notes
[slurm]/[pbs]fall back to SLURM inCluster::fillData(), soKind::Noneis unreachable on unattended installs.apt install.Fixes #63