diff --git a/CHANGELOG.md b/CHANGELOG.md index 5cd35b2..54003f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,21 @@ and this project adheres to ## [Unreleased] +### Changed + +- The agent briefing and the `hpc-compute`, `hpc-storage` and `slurm-batch` + skills say two more things. `/tmp` is the node's own disk — `$TMPDIR` and + the scratchpad directory an agent is told to use sit under it — so a + script staged there in a session does not exist on the node an `srun` + lands on, a failure that kept recurring as `No such file or directory`; + whatever crosses the session/job line goes on the shared filesystem + instead, under `~/scratch//` on Bodhi and + `/scratch/alpine/$USER//` on Alpine, and the briefing names the + path for the cluster it runs on. And a workflow controller — snakemake, + nextflow — is submitted with `sbatch` as a job of its own, so it outlives + the session rather than dying with it and taking the rest of the pipeline + along. + ### Fixed - The monitor panel showed only `gpu0` and `gpu1` on a four-GPU node: the diff --git a/crates/sint/src/commands/agent_context.rs b/crates/sint/src/commands/agent_context.rs index 3ec88de..3f3c133 100644 --- a/crates/sint/src/commands/agent_context.rs +++ b/crates/sint/src/commands/agent_context.rs @@ -13,6 +13,16 @@ //! resource numbers below are reported but never exported as environment //! variables — a `SINTERACTIVE_CPUS` in the environment is an invitation to //! run `make -j` here. +//! +//! Two more rules are here because agents kept getting them wrong. `/tmp` is +//! node-local, and the scratchpad directory an agent is told to use sits +//! under it, so a script staged there is "No such file" on the node an +//! `srun` lands on — what crosses that line goes on the shared filesystem, +//! and the briefing names the cluster's scratch for it. And a workflow +//! controller (snakemake, nextflow) is submitted as a job of its own, so it +//! outlives the session instead of dying with it. + +use std::path::{Path, PathBuf}; use anyhow::{anyhow, Result}; use sint_core::quota::{self, kb_to_size}; @@ -81,6 +91,11 @@ pub fn briefing(ctx: &Ctx) -> Result { let node = &row.node; let partition = &row.partition; + let home = std::env::var_os("HOME") + .map(PathBuf::from) + .unwrap_or_else(|| PathBuf::from("~")); + let scratch = shared_scratch(&home, &std::env::var("USER").unwrap_or_default()); + let scratch = scratch.display(); Ok(format!( r#"You are inside an sinteractive zellij session on a compute node. {ident} on {node}, partition {partition} — {res} @@ -112,6 +127,22 @@ work — send it to a compute partition instead. SLURM_* is stripped from this session, so srun and salloc create their own allocations rather than steps of this job — and work in them is bounded by their own -t, not by the budget above. +/tmp is this node's own disk, and so is everything under it — $TMPDIR and +the scratchpad directory you were told to use for temporary files included. +An srun or salloc lands on some other node, which sees none of it, and this +session sees nothing a job leaves on its /tmp. So before staging anything on +/tmp, ask who has to read it: whatever crosses that line — a script the job +runs, inputs it reads, output you want back — goes on the shared filesystem, +under {scratch}// named for the task. A job's own intermediates, +written and consumed inside one allocation, still belong on that node's /tmp. + +A workflow controller — snakemake, nextflow, anything that sits for hours +submitting jobs — is itself submitted with sbatch, as a job of its own with +a few CPUs and a long -t, and drives the real work as Slurm jobs (snakemake's +slurm executor, nextflow's slurm executor). Run in this session, or in an +srun held open from it, it dies with the session and the rest of the pipeline +with it; as a job it is bounded by nothing but its own -t. + Re-check this session with `sinteractive status --json` before long work; the number above was read when this briefing was generated, and a walltime can be changed underneath you. @@ -124,3 +155,29 @@ the warning clears immediately rather than up to ten minutes later. "# )) } + +/// The shared-filesystem scratch the briefing names for files that have to +/// cross between nodes: Alpine's `/scratch/alpine/$USER` where that exists, +/// else `~/scratch` — on a one-filesystem cluster such as Bodhi, home *is* +/// the shared filesystem, and `~/scratch//` is where throwaway +/// working files go. +fn shared_scratch(home: &Path, user: &str) -> PathBuf { + let alpine = Path::new("/scratch/alpine").join(user); + if alpine.is_dir() { + return alpine; + } + home.join("scratch") +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn scratch_is_under_home_where_there_is_no_alpine_scratch() { + assert_eq!( + shared_scratch(Path::new("/beevol/home/x"), "nobody-has-this-scratch-dir"), + PathBuf::from("/beevol/home/x/scratch") + ); + } +} diff --git a/crates/sint/src/commands/mcp.rs b/crates/sint/src/commands/mcp.rs index 995a9c3..4c3e406 100644 --- a/crates/sint/src/commands/mcp.rs +++ b/crates/sint/src/commands/mcp.rs @@ -900,7 +900,14 @@ mod tests { } fn write_state(state: &StateDir, job_id: u64, remaining: Option) { - let now = now_epoch(); + write_state_at(state, job_id, remaining, now_epoch()); + } + + /// `write_state` stamped with a caller-chosen clock, so a test that + /// asserts on the exact remaining seconds can observe with the same + /// `now` it wrote — a second ticking over between the two reads + /// `remaining - 1`, which is what made the walltime test flake in CI. + fn write_state_at(state: &StateDir, job_id: u64, remaining: Option, now: i64) { state .write_state(&StateFile { job_id, @@ -1029,39 +1036,40 @@ mod tests { #[test] fn synthetic_walltime_events_from_the_state_file() { let (_tmp, state) = dir(); - write_state(&state, 7, Some(5000)); - let mut watch = WalltimeWatch::new(&state, 7, T, now_epoch()); + let now = now_epoch(); + write_state_at(&state, 7, Some(5000), now); + let mut watch = WalltimeWatch::new(&state, 7, T, now); assert_eq!(watch.last, Some(5000)); // Still plenty: nothing. - write_state(&state, 7, Some(3000)); - assert_eq!(watch.observe(&state, 7, now_epoch()), None); + write_state_at(&state, 7, Some(3000), now); + assert_eq!(watch.observe(&state, 7, now), None); // Crossing the warning line. - write_state(&state, 7, Some(1800)); - let ev = watch.observe(&state, 7, now_epoch()).expect("warn"); + write_state_at(&state, 7, Some(1800), now); + let ev = watch.observe(&state, 7, now).expect("warn"); assert_eq!(ev.kind, "walltime_warn"); assert_eq!(ev.remaining_seconds, Some(1800)); assert!(ev.to_value()["synthetic"].as_bool().unwrap()); // Under it already: no repeat. - write_state(&state, 7, Some(1700)); - assert_eq!(watch.observe(&state, 7, now_epoch()), None); + write_state_at(&state, 7, Some(1700), now); + assert_eq!(watch.observe(&state, 7, now), None); // Crossing red. - write_state(&state, 7, Some(599)); + write_state_at(&state, 7, Some(599), now); assert_eq!( - watch.observe(&state, 7, now_epoch()).map(|e| e.kind), + watch.observe(&state, 7, now).map(|e| e.kind), Some("walltime_red") ); // The session ends: the file goes, once. fs::remove_file(state.state_file(7)).unwrap(); assert_eq!( - watch.observe(&state, 7, now_epoch()).map(|e| e.kind), + watch.observe(&state, 7, now).map(|e| e.kind), Some("session_ended") ); - assert_eq!(watch.observe(&state, 7, now_epoch()), None); + assert_eq!(watch.observe(&state, 7, now), None); } #[test] diff --git a/crates/sint/tests/reporting.rs b/crates/sint/tests/reporting.rs index bd32675..ab727fd 100644 --- a/crates/sint/tests/reporting.rs +++ b/crates/sint/tests/reporting.rs @@ -481,9 +481,18 @@ fn agent_context_briefing() { "(shown in full by `sinteractive status`)", "`sinteractive quota --check`", "be changed underneath you.\n\nStorage quota, while exceeded,", + "/tmp is this node's own disk", + "is itself submitted with sbatch", ] { assert!(out.contains(needle), "missing {needle:?} in:\n{out}"); } + // The scratch named for cross-node files is the cluster's: with no + // /scratch/alpine/$USER on this host, that is ~/scratch. + let scratch = format!( + "under {}/scratch// named for the task", + fx.home_dir().display() + ); + assert!(out.contains(&scratch), "missing {scratch:?} in:\n{out}"); assert!(!out.contains("--status"), "{out}"); assert!(!out.contains("--check-quota"), "{out}"); assert!(!out.contains("OVER STORAGE QUOTA"), "{out}"); diff --git a/docs/scripting.md b/docs/scripting.md index a6a330a..9c6883b 100644 --- a/docs/scripting.md +++ b/docs/scripting.md @@ -194,7 +194,9 @@ never fed the other one's partitions, paths, and quotas. `hpc-compute` covers cluster etiquette: neither the login node nor an sinteractive session is a compute target, real work goes into an allocation -sized for it, reuse sessions rather than piling them up, check the time budget +sized for it, nothing on a node's `/tmp` crosses into that allocation, a +workflow controller is submitted as a job of its own so it outlives the +session, reuse sessions rather than piling them up, check the time budget before long jobs, observe a session with `peek`/`send`, and clean up. `slurm-discovery` covers finding out what the cluster offers rather than @@ -207,8 +209,9 @@ cluster so the survey is run once rather than every session. `hpc-storage` covers where data goes, on both clusters this tool runs on. On Bodhi, `/beevol` is one shared BeeGFS mount and the compute node's `/tmp` is -a local disk, so inputs are read from the former and scratch is written to -the latter and cleaned up on exit. On Alpine (CU Boulder), the layout is +a local disk — that node's alone — so inputs are read from the former, +a job's scratch is written to the latter and cleaned up on exit, and what has +to cross between nodes goes under `~/scratch//`. On Alpine (CU Boulder), the layout is tiered the other way around: a 2 GB `/home` that nothing may be written to, a small backed-up `/projects`, and a huge purged `/scratch/alpine` parallel filesystem where all work runs. It also warns that `du` on a home directory @@ -223,9 +226,10 @@ shell, and never `pip install` into the system Python. `slurm-batch` covers work that is per-sample rather than a single command: `sbatch` scripts, arrays and why to throttle them with `%N`, dependency -chains, and using `sacct` to size the next run from what the last one actually -used — noting that `MaxRSS` lives on the step rows, where `sacct -X` will not -show it. +chains, submitting a snakemake or nextflow controller as a job of its own so +it outlives the session, and using `sacct` to size the next run from what the +last one actually used — noting that `MaxRSS` lives on the step rows, where +`sacct -X` will not show it. `git-workflow` covers the git conventions, and is about the repository open in the session rather than the cluster: semantic versioning with annotated diff --git a/skills/hpc-compute/SKILL.md b/skills/hpc-compute/SKILL.md index 871962e..e216b71 100644 --- a/skills/hpc-compute/SKILL.md +++ b/skills/hpc-compute/SKILL.md @@ -103,6 +103,33 @@ restrict which accounts and QOS may submit, so the right `-p` can still be rejected — the `slurm-discovery` skill covers mapping that out, and reading the reason when a job is refused or stuck `PENDING`. +### Nothing on `/tmp` crosses into an allocation + +`/tmp` is the node's own disk, and so is everything under it — `$TMPDIR`, +and the scratchpad directory an agent is told to use for temporary files. +An `srun` or `salloc` lands on some other node, which sees none of it, and +the session sees nothing a job leaves on *its* `/tmp`. The failure reads as +`No such file or directory` for a script written a minute ago. + +Before staging anything on `/tmp`, ask who has to read it. Whatever crosses +the session/job line — a script the job runs, inputs it reads, output you +want back — goes on the shared filesystem, in a directory named for the +task: `~/scratch//` on Bodhi, `/scratch/alpine/$USER//` on +Alpine (the briefing from `sinteractive claude context` names the one for +the cluster it runs on). What a job writes and consumes inside one +allocation still belongs on that node's `/tmp` — `hpc-storage` has the +pattern. + +### A workflow controller is a job, not a session + +`snakemake`, `nextflow` and their kind sit for hours submitting jobs. Run in +the session, or in an `srun` held open from it, the controller dies with the +session — walltime, a `scancel`, a maintenance window — and the rest of the +pipeline with it. Submit it with `sbatch` as a job of its own, so its +lifetime is its own `-t` and nothing else, and have it drive the real work +as Slurm jobs rather than running rules inline. `slurm-batch` has the +script. + ### Check for reservations before asking for walltime **A job asking for more walltime than remains before a maintenance diff --git a/skills/hpc-storage/SKILL.md b/skills/hpc-storage/SKILL.md index 72d692a..75ad064 100644 --- a/skills/hpc-storage/SKILL.md +++ b/skills/hpc-storage/SKILL.md @@ -36,8 +36,15 @@ hundreds of gigabytes to shared storage — that is a decision about other people's work, not just theirs. If they are already over quota, say so before starting rather than after the writes fail. -**Final artifacts go on storage every node can see** — anything staged on a -node-local disk vanishes from the next job's point of view. +**Only the shared filesystem is visible from more than one node.** `/tmp` +is the node's own disk — `$TMPDIR` and an agent's scratchpad directory are +under it — so a file written there in a session does not exist on the node +an `srun` lands on, and a job's `/tmp` output is gone from the session's +point of view. Before staging on `/tmp`, ask who has to read it: whatever +crosses that line — a script for a job, its inputs, output wanted back, +final artifacts — goes on the cluster's scratch, in a directory named for +the task (each cluster's file says where). A job's own intermediates, made +and consumed inside one allocation, are what node-local disk is for. Sizing an allocation for the job that does the writing is the `hpc-compute` skill; finding out which partition you may use is `slurm-discovery`. diff --git a/skills/hpc-storage/alpine.md b/skills/hpc-storage/alpine.md index cb43988..025557c 100644 --- a/skills/hpc-storage/alpine.md +++ b/skills/hpc-storage/alpine.md @@ -40,8 +40,12 @@ notebooks, environments, and final small outputs — never for the working set of a running pipeline. Node-local `/tmp` on a compute node is only ~63 GB — fine for a tool's small -temp files, too small for genomics intermediates. The working directory for -a job is scratch. +temp files, too small for genomics intermediates. It is also that node's +alone: a file written to `/tmp` in an sinteractive session — `$TMPDIR` and +an agent's scratchpad directory are under it — does not exist on the node an +`srun` lands on. The working directory for a job, and anything a job has to +read or you have to read back from it, is scratch: +`/scratch/alpine/$USER//`, named for the task. ## Checking space diff --git a/skills/hpc-storage/bodhi.md b/skills/hpc-storage/bodhi.md index 3853c49..71aa76d 100644 --- a/skills/hpc-storage/bodhi.md +++ b/skills/hpc-storage/bodhi.md @@ -48,8 +48,15 @@ cp "$work"/out.bam /beevol/home/$USER/results/ The `trap` matters: `/tmp` on a shared node already has a couple of thousand entries, and an uncleaned job directory sits there until someone notices. -Keep the *final* artifacts on `/beevol` — `/tmp` is node-local, so the next -job in the pipeline probably lands somewhere else and cannot see it. +That pattern holds *inside* one allocation. Across allocations `/tmp` is +worthless: it is that node's disk, so the next job in the pipeline lands +somewhere else and cannot see it, and nothing written to `/tmp` in an +sinteractive session — `$TMPDIR` and an agent's scratchpad directory are +under it — exists on the node an `srun` from that session lands on. What has +to cross that line goes on `/beevol`: final artifacts under the repo's +`results/`, and a task's working files — a script for a job to run, its +inputs, output you want back — under `~/scratch//`, named for the +task so the next session finds it and a cleanup pass knows what it was. ## Checking space diff --git a/skills/slurm-batch/SKILL.md b/skills/slurm-batch/SKILL.md index 451eaa4..0c7802b 100644 --- a/skills/slurm-batch/SKILL.md +++ b/skills/slurm-batch/SKILL.md @@ -95,6 +95,44 @@ be satisfied — the job it waits on failed — is **killed rather than left pending forever**. A vanished downstream job usually means an upstream failure, so check that first with `sacct` rather than resubmitting. +## A workflow controller + +`snakemake` and `nextflow` are schedulers of their own: one long-lived +process that submits a job per rule and waits for it. That process is +*itself* a job — never something run in an sinteractive session or in an +`srun` held open from one, where it dies with the session and takes the +rest of the pipeline with it. `sbatch` it with a couple of CPUs, a little +memory, and walltime that covers the whole run: + +```bash +#!/usr/bin/env bash +#SBATCH --job-name=smk-rnaseq +#SBATCH --comment=smk-rnaseq +#SBATCH --partition=rna +#SBATCH --cpus-per-task=2 +#SBATCH --mem=8G +#SBATCH --time=2-00:00:00 +#SBATCH --output=logs/%x-%j.out +set -euo pipefail + +cd /beevol/home/$USER/devel/proj # Snakefile, config and workdir on the shared filesystem +snakemake --executor slurm --jobs 50 +``` + +Snakemake's `slurm` executor (`snakemake-executor-plugin-slurm`) submits +each rule as its own job, sized from the rule's `resources:` — +`slurm_partition`, `mem_mb`, `runtime` — or a `--default-resources` line; +nextflow does the same with `process.executor = 'slurm'` in its config. The +controller's allocation holds only the controller, so what it needs is +walltime, not CPUs: past the `normal` QOS's 3-day ceiling on Bodhi add +`--qos=long`, and on Alpine use `cpu-long` for anything over 24h. Follow it +with `squeue --me`; `scancel` the controller to stop the pipeline, then check +the queue for rule jobs it had already submitted. + +Nothing staged on a session's `/tmp` reaches the controller or its jobs — +the Snakefile, config and working directory sit on the shared filesystem +(`hpc-compute` covers that boundary). + ## Right-size from what actually happened Run one sample, then look at what it used before committing to hundreds: