diff --git a/CHANGELOG.md b/CHANGELOG.md index 1852c9c..e27eea3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,32 @@ and this project adheres to ## [Unreleased] +### Added + +- Two skills that run as forked subagents on a cheaper model, so the + trivial end of a task does not replay the main conversation at the main + model's price. `land` (`model: sonnet`) commits the finished work with a + short Conventional Commit, pushes, opens the pull request and watches CI; + `job-watch` (`model: haiku`) waits for Slurm jobs and reports how they + ended — state, exit code, elapsed, peak memory against the request, the + log tail. The briefing and the MCP server's instructions tell the agent + to hand waits, status checks and landing off to them. + +### Changed + +- `git-workflow` is shorter and asks for short commit messages: most + commits are the subject line alone, and a body is two or three lines on + a why the diff cannot say, never a narrative. Releases tag the merge + commit on `main` after the pull request lands. +- `hpc-compute` is half its length. The sinteractive CLI walkthrough it + carried — `list`, `session ensure`, the state file, `peek`/`send` — is + what the MCP server now exposes as tools, so the skill keeps the rules + and a five-line reference; the `/tmp` and controller sections point at + `hpc-storage` and `slurm-batch` instead of repeating them. The + per-cluster storage files tell the `/tmp` story once, and every skill's + description, which sits in the system prompt of every session, is + tightened. + ## [1.2.0] - 2026-09-05 ### Changed diff --git a/README.md b/README.md index a5fa837..3d5e086 100644 --- a/README.md +++ b/README.md @@ -370,8 +370,9 @@ sinteractive claude install # from any installed copy make claude-install # equivalent, from a checkout ``` -This installs the six skills (`hpc-compute`, `slurm-discovery`, `hpc-storage`, -`hpc-software`, `slurm-batch`, `git-workflow`) into `~/.claude/skills/`, then +This installs the eight skills (`hpc-compute`, `slurm-discovery`, `hpc-storage`, +`hpc-software`, `slurm-batch`, `git-workflow`, and the forked `land` and +`job-watch`, which run on a cheaper model) into `~/.claude/skills/`, then registers in your `settings.json` the four hooks (`sinteractive claude hook session-start` briefs the agent on the session it is in; `sinteractive claude hook prompt` warns when walltime is short; `sinteractive claude hook diff --git a/crates/sint/src/commands/agent_context.rs b/crates/sint/src/commands/agent_context.rs index 3f3c133..e752c92 100644 --- a/crates/sint/src/commands/agent_context.rs +++ b/crates/sint/src/commands/agent_context.rs @@ -21,6 +21,11 @@ //! and the briefing names the cluster's scratch for it. And a workflow //! controller (snakemake, nextflow) is submitted as a job of its own, so it //! outlives the session instead of dying with it. +//! +//! And one about cost: the wait for a job, a queue or quota check, and +//! landing a branch are trivial, so the briefing sends them to a cheaper +//! model (the forked `job-watch` and `land` skills, or a haiku subagent) +//! rather than letting them replay the whole conversation at full price. use std::path::{Path, PathBuf}; @@ -143,6 +148,13 @@ slurm executor, nextflow's slurm executor). Run in this session, or in an srun held open from it, it dies with the session and the rest of the pipeline with it; as a job it is bounded by nothing but its own -t. +Not every step needs the model you are running. Waiting on a job, checking +the queue or the quota, and landing a finished branch are trivial: fork them +onto a cheaper model — `/job-watch JOBID` waits and reports how a job ended, +`/land "why"` commits, pushes and opens the pull request — or delegate to a +subagent with model haiku. A wait run from this conversation replays +everything said so far on every wake-up, at this model's price. + Re-check this session with `sinteractive status --json` before long work; the number above was read when this briefing was generated, and a walltime can be changed underneath you. diff --git a/crates/sint/src/commands/mcp.rs b/crates/sint/src/commands/mcp.rs index 4c3e406..040b5b8 100644 --- a/crates/sint/src/commands/mcp.rs +++ b/crates/sint/src/commands/mcp.rs @@ -58,8 +58,10 @@ nodes. A session is an orchestration shell, not a compute target: it is a small with the shell the user is typing in, so editing, git and scheduler queries belong there and \ anything heavier gets its own srun/salloc allocation. Call session_status before long work to \ read the remaining walltime, and wait_for_event (which blocks until something happens) instead \ -of polling. peek reads a session's screen; send types into the user's live shell, so only do that \ -when asked."; +of polling. Waiting and status checks are trivial work: run them from a subagent on a cheaper \ +model (the job-watch skill, or an Agent call with model haiku) rather than the main \ +conversation, which replays everything said so far on every wake-up. peek reads a session's \ +screen; send types into the user's live shell, so only do that when asked."; /// Default and cap for `wait_for_event`'s timeout, in seconds. const WAIT_DEFAULT_SECS: u64 = 300; diff --git a/crates/sint/tests/mcp.rs b/crates/sint/tests/mcp.rs index 12ffea3..f1478a6 100644 --- a/crates/sint/tests/mcp.rs +++ b/crates/sint/tests/mcp.rs @@ -160,6 +160,7 @@ fn handshake_tools_and_the_documented_contracts() { "{instructions}" ); assert!(instructions.contains("wait_for_event"), "{instructions}"); + assert!(instructions.contains("cheaper model"), "{instructions}"); let tools = mcp.request("tools/list", json!({})); let mut names: Vec<&str> = tools["tools"] diff --git a/crates/sint/tests/reporting.rs b/crates/sint/tests/reporting.rs index ab727fd..4089ea6 100644 --- a/crates/sint/tests/reporting.rs +++ b/crates/sint/tests/reporting.rs @@ -483,6 +483,9 @@ fn agent_context_briefing() { "be changed underneath you.\n\nStorage quota, while exceeded,", "/tmp is this node's own disk", "is itself submitted with sbatch", + "Not every step needs the model you are running", + "`/job-watch JOBID`", + "`/land \"why\"`", ] { assert!(out.contains(needle), "missing {needle:?} in:\n{out}"); } diff --git a/docs/scripting.md b/docs/scripting.md index 9c6883b..e5a1a7a 100644 --- a/docs/scripting.md +++ b/docs/scripting.md @@ -184,9 +184,9 @@ are removed when their `hpc-*` successor is installed. Exit codes: 0 done, 1 no assets found, 2 a settings file was refused (everything else was still installed). -**Six [skills](https://code.claude.com/docs/en/skills)** teach agents how work +**Eight [skills](https://code.claude.com/docs/en/skills)** teach agents how work is done here. Skills load on demand from their descriptions, so an agent picks -up the one the task calls for rather than carrying all six. The three `hpc-*` +up the one the task calls for rather than carrying all eight. The three `hpc-*` skills go one step further: their SKILL.md holds the rules shared by both clusters this tool runs on and delegates the rest to an `alpine.md` or `bodhi.md` beside it, so the agent reads the system it is actually on and is @@ -196,8 +196,9 @@ never fed the other one's partitions, paths, and quotas. sinteractive session is a compute target, real work goes into an allocation sized for it, nothing on a node's `/tmp` crosses into that allocation, a workflow controller is submitted as a job of its own so it outlives the -session, reuse sessions rather than piling them up, check the time budget -before long jobs, observe a session with `peek`/`send`, and clean up. +session, the wait for a job is handed to a cheaper model, reuse sessions +rather than piling them up, check the time budget before long jobs, and +clean up. `slurm-discovery` covers finding out what the cluster offers rather than assuming it: what the partitions are and how big, which accounts and QOS you @@ -233,10 +234,18 @@ last one actually used — noting that `MaxRSS` lives on the step rows, where `git-workflow` covers the git conventions, and is about the repository open in the session rather than the cluster: semantic versioning with annotated -`vX.Y.Z` tags, Conventional Commit messages, one worktree per branch under -`.claude/worktrees/`, landing work through a pull request rather than +`vX.Y.Z` tags, Conventional Commit messages kept to a subject line, one +worktree per branch, landing work through a pull request rather than committing to `main`, and running the repo's own CI gates before pushing. +`land` and `job-watch` are the trivial ends of the two workflows above, and +run as forked subagents on a cheaper model — `context: fork` with +`model: sonnet` and `model: haiku` in their frontmatter — so that committing, +pushing and watching CI, or waiting for a Slurm job and reporting how it +ended, does not replay the main conversation at the main model's price. The +briefing and the MCP server's instructions tell the agent to hand those +steps off. + **`sinteractive claude context`** prints a briefing on the current session — job, node, partition, allocation size, walltime remaining, and the rules above. It exits 1 outside a session. Run it by hand to see exactly what an diff --git a/skills/git-workflow/SKILL.md b/skills/git-workflow/SKILL.md index 4a716dc..50b83b7 100644 --- a/skills/git-workflow/SKILL.md +++ b/skills/git-workflow/SKILL.md @@ -1,140 +1,82 @@ --- name: git-workflow -description: Git conventions for this user's repositories — semantic versioning, Conventional Commit messages, one worktree per branch, and landing work through pull requests. Use whenever a task involves branching, committing, opening or reviewing a pull request, cutting a release, tagging a version, or writing a changelog entry. +description: Git conventions for this user's repositories — a worktree per branch, short Conventional Commit messages, semver with annotated tags, and landing through a pull request. Use when branching, committing, opening a pull request, cutting a release, or writing a changelog entry. --- # Git workflow -These are the user's standing preferences. They apply to whatever repository -is open, not to any one project. Where a repository documents something -stricter of its own — a `CONTRIBUTING.md`, a release checklist — that wins. +Standing preferences for every repository. Where a repository documents +something stricter of its own — a `CONTRIBUTING.md`, a release checklist — +that wins. ## Branch in a worktree, never on `main` -One worktree per line of work, made with the `EnterWorktree` tool; prefer it -over `git worktree add` so the session's working directory follows the -worktree instead of being left behind in the main checkout. Where the -worktree lands is the tool's business, not yours: with sinteractive's -`worktree-create` hook registered (`sinteractive claude install`) it is on -the cluster's scratch filesystem — `/scratch/alpine/$USER/worktrees// -` on Alpine — because a worktree is a throwaway build tree and -`/projects` is the small, backed-up tier; elsewhere it is Claude Code's -stock `/.claude/worktrees/`. `git worktree list` says where. -Never symlink `.claude/worktrees` somewhere else: Claude Code refuses to -create a worktree through a symlink. - -The base commit comes from the `worktree.baseRef` setting: `fresh` (the -default) branches from `origin/`, so the work starts from what -is actually on the remote rather than from whatever the local checkout has -drifted to; `head` branches from local `HEAD`, for work that genuinely builds -on uncommitted local history. - -`.claude/worktrees/` is a byproduct of the workflow, not source. If the -repository does not already ignore it, add it to `.gitignore` — or to -`.git/info/exclude` when the ignore file is shared and the convention is not. -A worktree on scratch needs no ignoring; it is not inside the checkout. - -Leave with `ExitWorktree`: `keep` while the branch is still in flight, -`remove` once the pull request has merged. A worktree outliving its branch is -a checkout of something that no longer exists. - -Branch names are short kebab-case topics describing the change, reading much -the way the subject line does: `fix-stale-time-budget`, `ci-validate-shell`, -`install-claude-from-anywhere`. +One worktree per line of work, made with `EnterWorktree`. Where it lands is +the tool's business (`git worktree list` says; with sinteractive's hooks +registered it is on the cluster's scratch on Alpine). Never symlink +`.claude/worktrees` elsewhere — Claude Code refuses to create a worktree +through a symlink — and make sure the repository ignores it. Leave with +`ExitWorktree`: `keep` while the branch is in flight, `remove` once merged. -## Land through a pull request - -```bash -git push -u origin HEAD -gh pr create --fill # then edit the body to say why -gh pr checks --watch # let CI go green before merging -gh pr merge -``` +Branch names are short kebab-case topics: `fix-stale-time-budget`, +`ci-validate-shell`. -Nothing goes onto `main` directly — not a typo fix, not a version bump, not a -one-line revert. The pull request is where CI runs and where the reasoning is -recorded; a change that skips it has neither, and the gap only surfaces later, -when somebody asks why a line is the way it is. - -Never force-push a branch someone else may have checked out, and never rewrite -history that is already on the remote. - -## Conventional Commits +## Commit messages are short ``` type(scope): subject - -Body explaining why, wrapped at 72. - -BREAKING CHANGE: what callers must now do differently. ``` Types: `feat`, `fix`, `docs`, `chore`, `refactor`, `test`, `ci`, `perf`, -`build`, `revert`. The scope is optional and names the area touched -(`fix(nodes):`, `feat(claude):`); leave it off when the change is repo-wide. +`build`, `revert`. The scope is optional and names the area touched. The +subject is imperative, lowercase, no trailing period, under 72 characters, +and completes "this commit will …". -The subject is imperative, lowercase, no trailing period, and completes the -sentence "this commit will …". The body explains *why* — the failure mode -being fixed, the alternative that was rejected and what was wrong with it — -because the diff already says what changed and nothing else records the -reasoning. A mechanical change needs no body; a judgment call always does. +**Most commits are the subject line alone.** Add a body only when the diff +cannot say why — a non-obvious cause, a constraint that forced the shape — +and keep it to two or three lines. No narrative of what changed, what else +was tried, or how the problem was found. Mark a major bump with `!` before +the colon or a `BREAKING CHANGE:` footer. -Mark anything forcing a major version bump with either a `BREAKING CHANGE:` -footer or a `!` before the colon (`feat(api)!:`). +Match a repository whose history plainly follows another convention, and +never rewrite history that is on the remote. -Where a repository's recent history plainly follows a different convention, -match the repository rather than switching styles mid-log — and never rewrite -existing commits to conform. - -## Semantic versioning - -`MAJOR.MINOR.PATCH`: MAJOR when existing usage breaks, MINOR for -backwards-compatible additions, PATCH for fixes that change no interface. -Before 1.0.0 the guarantee shifts down a place — MINOR is where breaking -changes go, and users should expect them there. +## Land through a pull request -Tags are `v`-prefixed and **annotated**: +Nothing goes onto `main` directly — not a typo fix, not a version bump. +Before pushing, run what CI runs (`.github/workflows/*.yml`: usually a +formatter, a linter, tests); when those are real compute, give them their +own allocation (`hpc-compute`). ```bash -git tag -a v1.4.0 -m 'Release v1.4.0' +git push -u origin HEAD +gh pr create --fill # body: one to three sentences on why +gh pr checks --watch +gh pr merge --squash # once green ``` -A lightweight tag is a bare pointer with no tagger, date, or message, so a -release cut that way leaves no record of when it was made or by whom. -Annotate every one. +Committing, pushing, opening the PR and watching CI is mechanical: the +`land` skill does it in a forked subagent on a cheaper model. Invoke it with +a one-line why once the work and its checks are done, rather than spending +the main conversation on it. -## Releasing +## Versions and releases -**Find every place the version is written before changing any of them.** It is -routinely more than one: a `VERSION=` in a script, `pyproject.toml`, -`package.json`, a `DESCRIPTION`, the `.TH` line of a man page, a docs config. -Grep for the current version string across the repository and bump the whole -set in one commit — a stale copy is invisible until a user reports that -`--version` disagrees with the tag. +`MAJOR.MINOR.PATCH`: MAJOR when existing usage breaks, MINOR for additions, +PATCH for fixes. Before 1.0.0, breaking changes go in MINOR. -Then, for a repository keeping a changelog in Keep a Changelog form: rename -`## [Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD`, open a fresh empty -`[Unreleased]` above it, and update the comparison links at the foot of the -file — `[Unreleased]` moves to `compare/vX.Y.Z...HEAD`, and a new `[X.Y.Z]` -link points at `compare/vPREV...vX.Y.Z`. +To release: grep the current version string across the repository and bump +every copy in one commit (a `VERSION=`, `Cargo.toml`, `pyproject.toml`, the +man page's `.TH` line, a docs config). In a Keep a Changelog file, +`[Unreleased]` becomes `[X.Y.Z] - YYYY-MM-DD` with a fresh empty +`[Unreleased]` above it, and the comparison links at the foot move. Commit +as `chore(release): vX.Y.Z` and land it through a pull request like +anything else. Then tag the merge commit on `main` — annotated, never +lightweight, so the tag records who cut it and when: ```bash -git commit -m 'chore(release): v1.4.0' +git checkout main && git pull git tag -a v1.4.0 -m 'Release v1.4.0' git push --follow-tags gh release create v1.4.0 --generate-notes ``` - -A release is still a pull request. Tag the merge commit on `main`, not the -branch. - -## Run the repository's own checks before pushing - -Read `.github/workflows/*.yml` and run what CI runs, locally, first. The gates -are usually a linter, a formatter check, and a test suite, and they take -seconds by hand; discovering them from a red pull request costs a round trip -and leaves a failed run in the history for nothing. - -When those checks are heavy enough to be real compute — a full test suite, a -build — they belong in their own Slurm allocation rather than in the session -shell. See the `hpc-compute` skill. diff --git a/skills/hpc-compute/SKILL.md b/skills/hpc-compute/SKILL.md index e216b71..7ea0f79 100644 --- a/skills/hpc-compute/SKILL.md +++ b/skills/hpc-compute/SKILL.md @@ -1,6 +1,6 @@ --- name: hpc-compute -description: Run compute work on the Bodhi and Alpine (CU Boulder/CURC) HPC clusters. Use whenever a task involves builds, analyses, pipelines, simulations, or any CPU-, memory-, or GPU-heavy or long-running command — such work must run inside a Slurm allocation sized for it, never on the login node and never in an sinteractive session. Covers deciding which cluster and node you are on, getting an allocation with srun or salloc, choosing the cluster's partitions and QOS, managing sinteractive sessions, checking time budgets, and observing the user's interactive sessions. +description: Run compute on the Bodhi and Alpine (CU Boulder/CURC) clusters. Anything CPU-, memory- or GPU-heavy or long-running — builds, analyses, pipelines, simulations — goes in its own Slurm allocation, never on the login node or in an sinteractive session. Covers telling the clusters apart, srun and salloc, naming jobs, the /tmp boundary, maintenance reservations, sessions, and what to hand to a cheaper model. --- # Running compute work @@ -8,8 +8,7 @@ description: Run compute work on the Bodhi and Alpine (CU Boulder/CURC) HPC clus ## First: where am I? Two clusters share this skill. Detect which, and read **only that cluster's -file** in this skill's directory for partitions, QOS, and local hazards — -the other cluster's details are noise: +file** in this skill's directory for partitions, QOS and local hazards: ```bash [ -d /scratch/alpine ] && echo alpine || { [ -d /beevol ] && echo bodhi; } @@ -18,123 +17,76 @@ the other cluster's details are noise: - **Alpine** (CU Boulder / CURC) → read `alpine.md` next to this SKILL.md - **Bodhi** → read `bodhi.md` next to this SKILL.md -`$SINTERACTIVE_JOB_ID` set → you are inside an sinteractive zellij session on -a compute node. Unset → you are on the login node. - -**The rule is the same either way: that shell is for orchestration, not -compute.** Editing, git, `squeue`/`sinfo`, and other sub-CPU-minute commands -belong there. Everything heavier gets its own allocation, sized for the job. - -An sinteractive session is *not* a compute target, even though it lives on a -compute node. It is usually a small allocation shared with the shell the -user is actively typing in. Do not run heavy commands in one, and do not -`srun --overlap` into one — you would be competing with the user for a small -allocation. +`$SINTERACTIVE_JOB_ID` set → you are inside an sinteractive session on a +compute node; unset → the login node. **Either way that shell is for +orchestration, not compute.** Editing, git, `squeue`/`sinfo` and other +sub-CPU-minute commands belong there; everything heavier gets its own +allocation, sized for the job. A session is a small allocation shared with +the shell the user is typing in — never run heavy commands in one and never +`srun --overlap` into one. ## Run work in its own allocation -`SLURM_*` is stripped from an sinteractive session, so `srun` and `salloc` run -from inside one create their own allocations rather than steps of the -session's job. Both stream stdout/stderr back and propagate the command's exit -code. - -**Name every job in both fields.** Give each `srun` and `salloc` a short, -descriptive name and pass it twice — `-J NAME` and `--comment=NAME`, the same -value in both: - -```bash -squeue --me -o "%.10i %.20j %.20P %.10M %k" # %j is the name, %k the comment -``` - -Name the work, not the tool: `bwa-align`, not `job1` or `bash`. Left unset, -the job takes the command's basename and an empty comment, so a queue full of -`bash` and `uv` tells nobody sharing the partition what is running or why. -This mirrors how sinteractive tags its own sessions (`sint-NAME` as the job -name, `sinteractive:NAME` as the comment). +`SLURM_*` is stripped from a session, so `srun` and `salloc` run from inside +one create their own allocations rather than steps of the session's job. +Both stream stdout/stderr back and propagate the command's exit code. -Where a cluster's accounting does not store the comment (Bodhi's does not), -it is readable on a live job (`squeue`, `scontrol show job ID`) but comes -back empty from `sacct` history — the name is the half that survives there. -That is the reason to fill both rather than picking one. +**Name every job in both fields** — `-J NAME` and `--comment=NAME`, the same +short description of the work (`bwa-align`, not `bash`) — so a shared queue +says what is running and why (`squeue --me -o "%.10i %.20j %.20P %.10M %k"` +shows both). Bodhi's accounting drops the comment, so the name is what +survives in `sacct`; that is why both are filled. -**One-off job** — blocks until it finishes (partition and QOS come from the -cluster file): +**One-off** — blocks until it finishes (partition and QOS from the cluster +file): ```bash srun -p PART [--qos=QOS] -c 8 --mem 32G -t 1:00:00 -J make-test --comment=make-test -- make test ``` -Use the Bash tool's background mode for long ones; `srun` stays attached for -the duration. - -**Sustained or iterative work** — hold one allocation and reuse it, instead of -queueing separately for every command: - -```bash -salloc --no-shell -p PART [--qos=QOS] -c 32 --mem 96G -t 4:00:00 -J cargo-ci --comment=cargo-ci -# salloc: Granted job allocation 244001 <- on STDERR, not stdout -srun --overlap --jobid=244001 -- cargo build --release -srun --overlap --jobid=244001 -- cargo test -scancel 244001 # when done -``` - -Name and comment live on the allocation, so naming the `salloc` covers every -`srun --overlap` step run inside it. - -`salloc --no-shell` returns immediately. Capture the job id through `2>&1`: +**Sustained or iterative** — hold one allocation and reuse it: ```bash -ID=$(salloc --no-shell -p PART -c 32 --mem 96G -t 4:00:00 \ +ID=$(salloc --no-shell -p PART [--qos=QOS] -c 32 --mem 96G -t 4:00:00 \ -J cargo-ci --comment=cargo-ci 2>&1 | - sed -n 's/.*Granted job allocation \([0-9]*\).*/\1/p') + sed -n 's/.*Granted job allocation \([0-9]*\).*/\1/p') # announced on stderr +srun --overlap --jobid=$ID -- cargo build --release +srun --overlap --jobid=$ID -- cargo test +scancel $ID # always, when done ``` -Work inside such an allocation is bounded by that allocation's own `-t`, not -by the sinteractive session's walltime. The session dying still ends any -`srun` you are streaming from, so for anything long prefer the background Bash -mode over holding the stream, or check both budgets. - -**Always cancel an allocation you are done with.** A held `salloc` occupies -the nodes until its walltime expires. +Naming the `salloc` covers every `srun --overlap` step inside it. Work in +an allocation is bounded by its own `-t`, not the session's walltime — but +the session dying still ends any `srun` you are streaming from, so run long +ones in the Bash tool's background mode. -Request only what the task needs, and ask the user before requesting more -than a day of walltime or a whole node's worth of resources. Partitions can -restrict which accounts and QOS may submit, so the right `-p` can still be -rejected — the `slurm-discovery` skill covers mapping that out, and reading -the reason when a job is refused or stuck `PENDING`. +Request only what the task needs, and ask before requesting more than a day +of walltime or a whole node. Partitions restrict which accounts and QOS may +submit, so the right `-p` can still be rejected — `slurm-discovery` covers +mapping that out and reading why a job is refused or stuck `PENDING`. ### Nothing on `/tmp` crosses into an allocation -`/tmp` is the node's own disk, and so is everything under it — `$TMPDIR`, -and the scratchpad directory an agent is told to use for temporary files. -An `srun` or `salloc` lands on some other node, which sees none of it, and -the session sees nothing a job leaves on *its* `/tmp`. The failure reads as -`No such file or directory` for a script written a minute ago. - -Before staging anything on `/tmp`, ask who has to read it. Whatever crosses -the session/job line — a script the job runs, inputs it reads, output you -want back — goes on the shared filesystem, in a directory named for the -task: `~/scratch//` on Bodhi, `/scratch/alpine/$USER//` on -Alpine (the briefing from `sinteractive claude context` names the one for -the cluster it runs on). What a job writes and consumes inside one -allocation still belongs on that node's `/tmp` — `hpc-storage` has the -pattern. +`/tmp` is the node's own disk, and `$TMPDIR` and the agent scratchpad are +under it: a script staged there is `No such file or directory` on the node +an `srun` lands on, and a job's `/tmp` output is invisible from the session. +Whatever crosses the session/job line — a script the job runs, its inputs, +output wanted back — goes on the shared filesystem under a directory named +for the task: `~/scratch//` on Bodhi, `/scratch/alpine/$USER//` +on Alpine. `hpc-storage` has the full pattern. ### A workflow controller is a job, not a session `snakemake`, `nextflow` and their kind sit for hours submitting jobs. Run in the session, or in an `srun` held open from it, the controller dies with the -session — walltime, a `scancel`, a maintenance window — and the rest of the -pipeline with it. Submit it with `sbatch` as a job of its own, so its -lifetime is its own `-t` and nothing else, and have it drive the real work -as Slurm jobs rather than running rules inline. `slurm-batch` has the -script. +session and the rest of the pipeline with it. Submit it with `sbatch` as a +job of its own and have it drive the real work as Slurm jobs — `slurm-batch` +has the script. ### Check for reservations before asking for walltime **A job asking for more walltime than remains before a maintenance reservation does not fail — it is silently deferred to after the window.** -The job queues, looks normal, and the cost is invisible unless you check: ```bash scontrol show reservation # ACTIVE = on now; INACTIVE = scheduled @@ -142,126 +94,58 @@ srun --test-only -p PART -c 4 --mem 8G -t 21:00:00 -- true # srun: Job ... to start at