feat(claude): tell agents /tmp is node-local and controllers are jobs

CHANGELOG.md

@@ -8,6 +8,21 @@ and this project adheres to
 
 ## [Unreleased]
 
+### Changed
+
+- The agent briefing and the `hpc-compute`, `hpc-storage` and `slurm-batch`
+  skills say two more things. `/tmp` is the node's own disk — `$TMPDIR` and
+  the scratchpad directory an agent is told to use sit under it — so a
+  script staged there in a session does not exist on the node an `srun`
+  lands on, a failure that kept recurring as `No such file or directory`;
+  whatever crosses the session/job line goes on the shared filesystem
+  instead, under `~/scratch/<topic>/` on Bodhi and
+  `/scratch/alpine/$USER/<topic>/` on Alpine, and the briefing names the
+  path for the cluster it runs on. And a workflow controller — snakemake,
+  nextflow — is submitted with `sbatch` as a job of its own, so it outlives
+  the session rather than dying with it and taking the rest of the pipeline
+  along.
+
 ### Fixed
 
 - The monitor panel showed only `gpu0` and `gpu1` on a four-GPU node: the

crates/sint/src/commands/agent_context.rs

@@ -13,6 +13,16 @@
 //! resource numbers below are reported but never exported as environment
 //! variables — a `SINTERACTIVE_CPUS` in the environment is an invitation to
 //! run `make -j` here.
+//!
+//! Two more rules are here because agents kept getting them wrong. `/tmp` is
+//! node-local, and the scratchpad directory an agent is told to use sits
+//! under it, so a script staged there is "No such file" on the node an
+//! `srun` lands on — what crosses that line goes on the shared filesystem,
+//! and the briefing names the cluster's scratch for it. And a workflow
+//! controller (snakemake, nextflow) is submitted as a job of its own, so it
+//! outlives the session instead of dying with it.
+
+use std::path::{Path, PathBuf};
 
 use anyhow::{anyhow, Result};
 use sint_core::quota::{self, kb_to_size};
@@ -81,6 +91,11 @@ pub fn briefing(ctx: &Ctx) -> Result<String> {
 
     let node = &row.node;
     let partition = &row.partition;
+    let home = std::env::var_os("HOME")
+        .map(PathBuf::from)
+        .unwrap_or_else(|| PathBuf::from("~"));
+    let scratch = shared_scratch(&home, &std::env::var("USER").unwrap_or_default());
+    let scratch = scratch.display();
     Ok(format!(
         r#"You are inside an sinteractive zellij session on a compute node.
   {ident} on {node}, partition {partition} — {res}
@@ -112,6 +127,22 @@ work — send it to a compute partition instead. SLURM_* is stripped from this
 session, so srun and salloc create their own allocations rather than steps of
 this job — and work in them is bounded by their own -t, not by the budget above.
 
+/tmp is this node's own disk, and so is everything under it — $TMPDIR and
+the scratchpad directory you were told to use for temporary files included.
+An srun or salloc lands on some other node, which sees none of it, and this
+session sees nothing a job leaves on its /tmp. So before staging anything on
+/tmp, ask who has to read it: whatever crosses that line — a script the job
+runs, inputs it reads, output you want back — goes on the shared filesystem,
+under {scratch}/<topic>/ named for the task. A job's own intermediates,
+written and consumed inside one allocation, still belong on that node's /tmp.
+
+A workflow controller — snakemake, nextflow, anything that sits for hours
+submitting jobs — is itself submitted with sbatch, as a job of its own with
+a few CPUs and a long -t, and drives the real work as Slurm jobs (snakemake's
+slurm executor, nextflow's slurm executor). Run in this session, or in an
+srun held open from it, it dies with the session and the rest of the pipeline
+with it; as a job it is bounded by nothing but its own -t.
+
 Re-check this session with `sinteractive status --json` before long work;
 the number above was read when this briefing was generated, and a walltime can
 be changed underneath you.
@@ -124,3 +155,29 @@ the warning clears immediately rather than up to ten minutes later.
 "#
     ))
 }
+
+/// The shared-filesystem scratch the briefing names for files that have to
+/// cross between nodes: Alpine's `/scratch/alpine/$USER` where that exists,
+/// else `~/scratch` — on a one-filesystem cluster such as Bodhi, home *is*
+/// the shared filesystem, and `~/scratch/<topic>/` is where throwaway
+/// working files go.
+fn shared_scratch(home: &Path, user: &str) -> PathBuf {
+    let alpine = Path::new("/scratch/alpine").join(user);
+    if alpine.is_dir() {
+        return alpine;
+    }
+    home.join("scratch")
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn scratch_is_under_home_where_there_is_no_alpine_scratch() {
+        assert_eq!(
+            shared_scratch(Path::new("/beevol/home/x"), "nobody-has-this-scratch-dir"),
+            PathBuf::from("/beevol/home/x/scratch")
+        );
+    }
+}

crates/sint/src/commands/mcp.rs

@@ -900,7 +900,14 @@ mod tests {
     }
 
     fn write_state(state: &StateDir, job_id: u64, remaining: Option<i64>) {
-        let now = now_epoch();
+        write_state_at(state, job_id, remaining, now_epoch());
+    }
+
+    /// `write_state` stamped with a caller-chosen clock, so a test that
+    /// asserts on the exact remaining seconds can observe with the same
+    /// `now` it wrote — a second ticking over between the two reads
+    /// `remaining - 1`, which is what made the walltime test flake in CI.
+    fn write_state_at(state: &StateDir, job_id: u64, remaining: Option<i64>, now: i64) {
         state
             .write_state(&StateFile {
                 job_id,
@@ -1029,39 +1036,40 @@ mod tests {
     #[test]
     fn synthetic_walltime_events_from_the_state_file() {
         let (_tmp, state) = dir();
-        write_state(&state, 7, Some(5000));
-        let mut watch = WalltimeWatch::new(&state, 7, T, now_epoch());
+        let now = now_epoch();
+        write_state_at(&state, 7, Some(5000), now);
+        let mut watch = WalltimeWatch::new(&state, 7, T, now);
         assert_eq!(watch.last, Some(5000));
 
         // Still plenty: nothing.
-        write_state(&state, 7, Some(3000));
-        assert_eq!(watch.observe(&state, 7, now_epoch()), None);
+        write_state_at(&state, 7, Some(3000), now);
+        assert_eq!(watch.observe(&state, 7, now), None);
 
         // Crossing the warning line.
-        write_state(&state, 7, Some(1800));
-        let ev = watch.observe(&state, 7, now_epoch()).expect("warn");
+        write_state_at(&state, 7, Some(1800), now);
+        let ev = watch.observe(&state, 7, now).expect("warn");
         assert_eq!(ev.kind, "walltime_warn");
         assert_eq!(ev.remaining_seconds, Some(1800));
         assert!(ev.to_value()["synthetic"].as_bool().unwrap());
 
         // Under it already: no repeat.
-        write_state(&state, 7, Some(1700));
-        assert_eq!(watch.observe(&state, 7, now_epoch()), None);
+        write_state_at(&state, 7, Some(1700), now);
+        assert_eq!(watch.observe(&state, 7, now), None);
 
         // Crossing red.
-        write_state(&state, 7, Some(599));
+        write_state_at(&state, 7, Some(599), now);
         assert_eq!(
-            watch.observe(&state, 7, now_epoch()).map(|e| e.kind),
+            watch.observe(&state, 7, now).map(|e| e.kind),
             Some("walltime_red")
         );
 
         // The session ends: the file goes, once.
         fs::remove_file(state.state_file(7)).unwrap();
         assert_eq!(
-            watch.observe(&state, 7, now_epoch()).map(|e| e.kind),
+            watch.observe(&state, 7, now).map(|e| e.kind),
             Some("session_ended")
         );
-        assert_eq!(watch.observe(&state, 7, now_epoch()), None);
+        assert_eq!(watch.observe(&state, 7, now), None);
     }
 
     #[test]

crates/sint/tests/reporting.rs

@@ -481,9 +481,18 @@ fn agent_context_briefing() {
         "(shown in full by `sinteractive status`)",
         "`sinteractive quota --check`",
         "be changed underneath you.\n\nStorage quota, while exceeded,",
+        "/tmp is this node's own disk",
+        "is itself submitted with sbatch",
     ] {
         assert!(out.contains(needle), "missing {needle:?} in:\n{out}");
     }
+    // The scratch named for cross-node files is the cluster's: with no
+    // /scratch/alpine/$USER on this host, that is ~/scratch.
+    let scratch = format!(
+        "under {}/scratch/<topic>/ named for the task",
+        fx.home_dir().display()
+    );
+    assert!(out.contains(&scratch), "missing {scratch:?} in:\n{out}");
     assert!(!out.contains("--status"), "{out}");
     assert!(!out.contains("--check-quota"), "{out}");
     assert!(!out.contains("OVER STORAGE QUOTA"), "{out}");

docs/scripting.md

@@ -194,7 +194,9 @@ never fed the other one's partitions, paths, and quotas.
 
 `hpc-compute` covers cluster etiquette: neither the login node nor an
 sinteractive session is a compute target, real work goes into an allocation
-sized for it, reuse sessions rather than piling them up, check the time budget
+sized for it, nothing on a node's `/tmp` crosses into that allocation, a
+workflow controller is submitted as a job of its own so it outlives the
+session, reuse sessions rather than piling them up, check the time budget
 before long jobs, observe a session with `peek`/`send`, and clean up.
 
 `slurm-discovery` covers finding out what the cluster offers rather than
@@ -207,8 +209,9 @@ cluster so the survey is run once rather than every session.
 
 `hpc-storage` covers where data goes, on both clusters this tool runs on. On
 Bodhi, `/beevol` is one shared BeeGFS mount and the compute node's `/tmp` is
-a local disk, so inputs are read from the former and scratch is written to
-the latter and cleaned up on exit. On Alpine (CU Boulder), the layout is
+a local disk — that node's alone — so inputs are read from the former,
+a job's scratch is written to the latter and cleaned up on exit, and what has
+to cross between nodes goes under `~/scratch/<topic>/`. On Alpine (CU Boulder), the layout is
 tiered the other way around: a 2 GB `/home` that nothing may be written to,
 a small backed-up `/projects`, and a huge purged `/scratch/alpine` parallel
 filesystem where all work runs. It also warns that `du` on a home directory
@@ -223,9 +226,10 @@ shell, and never `pip install` into the system Python.
 
 `slurm-batch` covers work that is per-sample rather than a single command:
 `sbatch` scripts, arrays and why to throttle them with `%N`, dependency
-chains, and using `sacct` to size the next run from what the last one actually
-used — noting that `MaxRSS` lives on the step rows, where `sacct -X` will not
-show it.
+chains, submitting a snakemake or nextflow controller as a job of its own so
+it outlives the session, and using `sacct` to size the next run from what the
+last one actually used — noting that `MaxRSS` lives on the step rows, where
+`sacct -X` will not show it.
 
 `git-workflow` covers the git conventions, and is about the repository open in
 the session rather than the cluster: semantic versioning with annotated

skills/hpc-compute/SKILL.md

@@ -103,6 +103,33 @@ restrict which accounts and QOS may submit, so the right `-p` can still be
 rejected — the `slurm-discovery` skill covers mapping that out, and reading
 the reason when a job is refused or stuck `PENDING`.
 
+### Nothing on `/tmp` crosses into an allocation
+
+`/tmp` is the node's own disk, and so is everything under it — `$TMPDIR`,
+and the scratchpad directory an agent is told to use for temporary files.
+An `srun` or `salloc` lands on some other node, which sees none of it, and
+the session sees nothing a job leaves on *its* `/tmp`. The failure reads as
+`No such file or directory` for a script written a minute ago.
+
+Before staging anything on `/tmp`, ask who has to read it. Whatever crosses
+the session/job line — a script the job runs, inputs it reads, output you
+want back — goes on the shared filesystem, in a directory named for the
+task: `~/scratch/<topic>/` on Bodhi, `/scratch/alpine/$USER/<topic>/` on
+Alpine (the briefing from `sinteractive claude context` names the one for
+the cluster it runs on). What a job writes and consumes inside one
+allocation still belongs on that node's `/tmp` — `hpc-storage` has the
+pattern.
+
+### A workflow controller is a job, not a session
+
+`snakemake`, `nextflow` and their kind sit for hours submitting jobs. Run in
+the session, or in an `srun` held open from it, the controller dies with the
+session — walltime, a `scancel`, a maintenance window — and the rest of the
+pipeline with it. Submit it with `sbatch` as a job of its own, so its
+lifetime is its own `-t` and nothing else, and have it drive the real work
+as Slurm jobs rather than running rules inline. `slurm-batch` has the
+script.
+
 ### Check for reservations before asking for walltime
 
 **A job asking for more walltime than remains before a maintenance

skills/hpc-storage/SKILL.md

@@ -36,8 +36,15 @@ hundreds of gigabytes to shared storage — that is a decision about other
 people's work, not just theirs. If they are already over quota, say so
 before starting rather than after the writes fail.
 
-**Final artifacts go on storage every node can see** — anything staged on a
-node-local disk vanishes from the next job's point of view.
+**Only the shared filesystem is visible from more than one node.** `/tmp`
+is the node's own disk — `$TMPDIR` and an agent's scratchpad directory are
+under it — so a file written there in a session does not exist on the node
+an `srun` lands on, and a job's `/tmp` output is gone from the session's
+point of view. Before staging on `/tmp`, ask who has to read it: whatever
+crosses that line — a script for a job, its inputs, output wanted back,
+final artifacts — goes on the cluster's scratch, in a directory named for
+the task (each cluster's file says where). A job's own intermediates, made
+and consumed inside one allocation, are what node-local disk is for.
 
 Sizing an allocation for the job that does the writing is the `hpc-compute`
 skill; finding out which partition you may use is `slurm-discovery`.

skills/hpc-storage/alpine.md

@@ -40,8 +40,12 @@ notebooks, environments, and final small outputs — never for the working set
 of a running pipeline.
 
 Node-local `/tmp` on a compute node is only ~63 GB — fine for a tool's small
-temp files, too small for genomics intermediates. The working directory for
-a job is scratch.
+temp files, too small for genomics intermediates. It is also that node's
+alone: a file written to `/tmp` in an sinteractive session — `$TMPDIR` and
+an agent's scratchpad directory are under it — does not exist on the node an
+`srun` lands on. The working directory for a job, and anything a job has to
+read or you have to read back from it, is scratch:
+`/scratch/alpine/$USER/<topic>/`, named for the task.
 
 ## Checking space
 

skills/hpc-storage/bodhi.md

@@ -48,8 +48,15 @@ cp "$work"/out.bam /beevol/home/$USER/results/
 The `trap` matters: `/tmp` on a shared node already has a couple of thousand
 entries, and an uncleaned job directory sits there until someone notices.
 
-Keep the *final* artifacts on `/beevol` — `/tmp` is node-local, so the next
-job in the pipeline probably lands somewhere else and cannot see it.
+That pattern holds *inside* one allocation. Across allocations `/tmp` is
+worthless: it is that node's disk, so the next job in the pipeline lands
+somewhere else and cannot see it, and nothing written to `/tmp` in an
+sinteractive session — `$TMPDIR` and an agent's scratchpad directory are
+under it — exists on the node an `srun` from that session lands on. What has
+to cross that line goes on `/beevol`: final artifacts under the repo's
+`results/`, and a task's working files — a script for a job to run, its
+inputs, output you want back — under `~/scratch/<topic>/`, named for the
+task so the next session finds it and a cleanup pass knows what it was.
 
 ## Checking space
 

skills/slurm-batch/SKILL.md

@@ -95,6 +95,44 @@ be satisfied — the job it waits on failed — is **killed rather than left
 pending forever**. A vanished downstream job usually means an upstream
 failure, so check that first with `sacct` rather than resubmitting.
 
+## A workflow controller
+
+`snakemake` and `nextflow` are schedulers of their own: one long-lived
+process that submits a job per rule and waits for it. That process is
+*itself* a job — never something run in an sinteractive session or in an
+`srun` held open from one, where it dies with the session and takes the
+rest of the pipeline with it. `sbatch` it with a couple of CPUs, a little
+memory, and walltime that covers the whole run:
+
+```bash
+#!/usr/bin/env bash
+#SBATCH --job-name=smk-rnaseq
+#SBATCH --comment=smk-rnaseq
+#SBATCH --partition=rna
+#SBATCH --cpus-per-task=2
+#SBATCH --mem=8G
+#SBATCH --time=2-00:00:00
+#SBATCH --output=logs/%x-%j.out
+set -euo pipefail
+
+cd /beevol/home/$USER/devel/proj      # Snakefile, config and workdir on the shared filesystem
+snakemake --executor slurm --jobs 50
+```
+
+Snakemake's `slurm` executor (`snakemake-executor-plugin-slurm`) submits
+each rule as its own job, sized from the rule's `resources:` —
+`slurm_partition`, `mem_mb`, `runtime` — or a `--default-resources` line;
+nextflow does the same with `process.executor = 'slurm'` in its config. The
+controller's allocation holds only the controller, so what it needs is
+walltime, not CPUs: past the `normal` QOS's 3-day ceiling on Bodhi add
+`--qos=long`, and on Alpine use `cpu-long` for anything over 24h. Follow it
+with `squeue --me`; `scancel` the controller to stop the pipeline, then check
+the queue for rule jobs it had already submitted.
+
+Nothing staged on a session's `/tmp` reaches the controller or its jobs —
+the Snakefile, config and working directory sit on the shared filesystem
+(`hpc-compute` covers that boundary).
+
 ## Right-size from what actually happened
 
 Run one sample, then look at what it used before committing to hundreds: