Skip to content

contributing.md

Latest commit

 

History

History
485 lines (347 loc) · 54.8 KB

contributing.md

File metadata and controls

485 lines (347 loc) · 54.8 KB
Raw
title Contributing
permalink /contributing
diataxis contributor
redirect_from
/contributing.md

Contributing to Jaiph

Contributor docs answer a narrow question: where changes belong, how to run the same checks CI runs, and which test layer should encode a behavior change.

At a high level, Jaiph is built as described in Architecture — loadModuleGraph → per-module validateModule + script emit via buildScriptsFromGraph / emitScriptsForModuleFromGraph, the same graph consumed by buildRuntimeGraph(graph), validate-only jaiph compile (collectDiagnostics), NodeWorkflowRuntime, artifact layout. Treat that page as authoritative for pipelines and boundaries; if anything here diverges from it or from the implementation, prefer architecture + source.

For language syntax, library usage, tooling setup, and grammar details, see Language, Install & switch versions, Grammar, and Your first run. For the *.test.jh language and test blocks, see Write & run tests.

Branching and pull requests

Development moves quickly and may include breaking changes. Two primary branches: main (stable) and nightly (latest).

  • If you want to fix a bug, point your PR to the main branch, and check if the issue has already been addressed in nightly.
  • If you are adding a new feature, submit your PR to nightly (and add or update tests in e2e/tests).
  • For non-trivial bugs or features, open an issue with enough context to prioritize and discuss before or alongside a PR.
  • AI-assisted pull requests are welcome as long as they include comprehensive tests (including in e2e/tests).

Installing from source

Use the local installer wrapper script in this repo:

./docs/install-from-local.sh

After install, verify:

jaiph --version
jaiph --help

The script builds the self-contained standalone binary via docs/install (npm ci when a lockfile is present, else npm install, plus npm run build:standalone, including uncommitted changes; the copy skips .git, node_modules, and .jaiph/runs) and installs dist/jaiph to ~/.local/bin by default (or JAIPH_BIN_DIR if set). It does not build the linux runner image.

To build a local ghcr.io/jaiphlang/jaiph-runtime (linux binaries + runtime/Dockerfile):

./docs/build-jaiph-dev-image.sh

From-source prerequisites: npm and Bun. The image script also needs Docker.

Developing in the repository

For day-to-day work on the compiler and CLI you usually stay inside the clone: install dev dependencies once, then build and run tests from npm scripts.

Prerequisites: Node.js 20.x and npm (matching .github/workflows/ci.yml). Bun is also required for npm run build:standalone and ./docs/install-from-local.sh; standalone cross-compiles run in .github/workflows/release.yml via oven-sh/setup-bun, not in the main ci.yml unit/E2E jobs. End-user installs from docs/install need curl, shasum / sha256sum, and minisign to verify the release signature. The installers also expect bash. A missing minisign aborts the install on every host, including CI. For a deliberate checksum-only install, set JAIPH_ALLOW_UNSIGNED=1, which skips signature verification. End-to-end tests are written in bash and are run by e2e/test_all.sh.

Typical commands (from the repo root):

Command What it runs
npm install Installs TypeScript and types (dev dependencies).
npm run build Runs npm run embed-assets (regenerates src/runtime/embedded-assets.ts from docs/jaiph-skill.md, and src/version.ts from package.json's version field), then tsc, then copies src/runtime → dist/src/runtime (kernel JS for the compiled CLI).
npm run embed-assets Runs tools/embed-assets.js to regenerate src/runtime/embedded-assets.ts and src/version.ts. Invoked automatically by npm run build and by npm install (via the prepare lifecycle script, so gitignored src/version.ts exists on a fresh checkout before tsc); run it standalone after editing embedded source files if you want to refresh the generated copies without a full rebuild. src/runtime/embedded-assets.test.ts fails CI if checked-in src/runtime/embedded-assets.ts drifts from docs/jaiph-skill.md.
npm run build:standalone npm run build, then copies dist/src/runtime → dist/runtime and runs bun build --compile ./src/cli.ts --outfile ./dist/jaiph. Requires Bun. The resulting dist/jaiph is fully self-contained — jaiph-skill.md is baked into the binary, and workflow launch self-spawns via the internal __workflow-runner argv marker, so the binary needs no sibling runtime/ or docs/ files and no node / npm on the host. The dist/runtime copy is kept for parity with the npm layout (Architecture — Distribution).
npm run arch:check Runs dependency-cruiser over src/ with .dependency-cruiser.cjs to enforce the Agent analyzability import graph: no cycles, the layer DAG (each layer imports only downward, including runtime ↛ cli, and runtime reuses compile only through the single public entry src/transpiler.ts), no-deep-imports-into-parse (code outside the parse package imports only the public entry src/parser.ts, never a src/parse/** internal), no-deep-imports-into-transpile (code outside the transpile package imports only the single public entry src/transpiler.ts, which re-exports the module-graph API, never a src/transpile/** internal), no-deep-imports-into-runtime (code outside the runtime package imports only a public entry, src/runtime/index.ts for production or src/runtime/testing.ts for named test seams, never a src/runtime/** internal), no-deep-imports-into-format (code outside the format package imports only the public entry src/format/index.ts, never a src/format/** internal), and no-cross-cli-slice-imports (a file in one CLI slice such as commands or serve imports another slice's private tree only through src/cli/shared/** or a lower-layer public entry). Reads the TypeScript sources directly, so it needs no build. Pre-existing violations are grandfathered in .dependency-cruiser-known-violations.json (passed via --ignore-known), so old violations are tracked while a new cycle, upward import, parse, transpile, runtime, or format deep import, or cross-CLI-slice import fails the check. A required CI step on the Compiler and unit tests job.
npm run lint Runs ESLint over src/ with eslint.config.mjs and --max-warnings 0 to enforce the Agent analyzability fan-out and file-size caps: import/max-dependencies at 8 runtime imports per file (type imports ignored) and max-lines at 400 non-blank, non-comment lines. Test files are out of scope. Most files that once exceeded a cap were split into sibling modules and now pass the caps with no override; the four largest remaining files keep a per-file override in eslint.config.mjs that turns off only the rule they break, each with a fresh justification, and the global cap is never raised, so a new violation fails the check. A required CI step on the Compiler and unit tests job.
npm test npm run clean, then npm run build, then the Node.js test runner with NODE_OPTIONS including --enable-source-maps and a large heap limit, on every file under dist/integration/ matching *.test.js, every file under dist/src/ matching *.test.js or *.acceptance.test.js (via find), scripts/build-registry.test.mjs, dist/test-infra/compiler-test-runner.js (txtar compiler tests), and dist/test-infra/golden-ast-runner.js (golden AST tests).
npm run test:compiler npm run build, then node --test on dist/test-infra/compiler-test-runner.js — runs txtar-based compiler test fixtures from test-fixtures/compiler-txtar/.
npm run test:golden-ast npm run build, then node --test on dist/test-infra/golden-ast-runner.js — runs golden AST tests from test-fixtures/golden-ast/. Use UPDATE_GOLDEN=1 npm run test:golden-ast to regenerate goldens after intentional parser changes.
npm run test:acceptance:compiler npm run build, then node --test with only *.acceptance.test.js files under dist/src/ — compiler acceptance tests without the full unit suite or E2E.
npm run test:acceptance:runtime bash ./e2e/test_all.sh only — same E2E driver as below without an implicit rebuild; run dist/ is up to date before running.
npm run test:acceptance npm run test:acceptance:compiler then npm run test:acceptance:runtime.
npm run test:e2e npm run build, then bash ./e2e/test_all.sh. Prefer this when you want a fresh dist/ before E2E. e2e::prepare_shared_context in e2e/lib/common.sh clears most inherited JAIPH_* variables before each test. Host-only. See E2E testing.
npm run test:samples npx playwright test — Playwright suite for the docs landing page (e2e/playwright/). Uses http://127.0.0.1:4000 (see playwright.config.ts); starts Jekyll via webServer or reuses one already on that port. Requires Playwright (npx playwright install chromium once).
npm run test:ci npm test followed by npm run test:e2e — useful before pushing when you want the full local picture.

Run a single Node test file after a build with e.g. node --test dist/src/parse/parse-core.test.js. The dist/ paths mirror the source layout under src/.

Workspace hygiene

The root .gitignore blocks common debug and temp directory patterns so they never reach version control:

Pattern Purpose
docker-*/ Leftover Docker debug/experiment directories
nested-*/ Nested-run debug directories
local-*/ Local debug directories
.tmp/ Temp build/debug directory (exact name)
.tmp*/ Temp build/debug directories (prefix)
QUEUE.md.tmp.* Stale queue temp files

If you create throwaway directories during development, use one of these prefixes so they are automatically ignored. To track a file that matches a blocked pattern, use git add -f.

Code philosophy

Jaiph's codebase is maintained by both humans and AI agents. Code should be easy to read, navigate, and modify for both — which means the same thing: straightforward, flat, and explicit.

Principles

  1. Plain functions with explicit arguments. Avoid classes and abstraction-heavy generics. If a file already uses generics, follow the local style and keep additions minimal. No visitor patterns or dependency injection. A function takes data in, returns data out, or pushes to an array. If you need shared context, pass a plain object — not a class instance.
  2. Flat control flow. If/else chains and switch statements, not multi-layer abstractions. An AI (or human) should be able to read a function top-to-bottom and understand what it does without jumping to 3 other files.
  3. Short files. No source file should exceed ~400 lines. If a file grows past that, split it into sibling files in the same directory — not a deeper module tree.
  4. No duplication across files. If the same logic exists in two places, extract it into one plain function and call it from both. Duplication means bug fixes get applied in one place and missed in the other.
  5. Minimal touch surface for new features. Adding a new step type, CLI command, or runtime feature should require changes in at most 3 files (type definition, parser/handler, emitter/consumer). If it requires more, the architecture needs refactoring first.
  6. No speculative abstractions. Don't add indirection, registries, or plugin systems to "support future extensibility." Add the simplest thing that works now. Refactor when the concrete need appears.
  7. Red flag: >100 lines of custom code. Before writing a large chunk, check if the library or existing codebase already provides it. If not, explain why the new code is necessary.

Testing philosophy

Jaiph uses several test layers. Each layer catches a different class of bug. Use the narrowest layer that covers the behavior you're verifying.

Layer Location What it catches When to use
Module tests src/**/*.test.ts (colocated) Bugs in pure functions (event parsing, param formatting, path resolution, config merging) The function is self-contained, takes input and returns output, no I/O
Compiler acceptance tests src/transpile/*.acceptance.test.ts (colocated) Cross-module compiler behavior: validation errors, resolution, and other cases that need a temp project tree or subprocess You need a deterministic error string, multi-file buildScripts, or behavior that does not fit a tiny golden snippet
Compiler golden tests src/transpile/compiler-golden.test.ts (colocated) Regressions in the parser, validation messages, and scripts-only extraction (buildScriptFiles in emit-script.ts) — expectations are inline in the test file You changed the parser, validator, or script extraction and need to lock an exact error string, extracted script shape, or corpus behavior
Trivia / formatter round-trip src/parse/trivia-ast-shape.test.ts, src/parse/trivia-grep.test.ts, src/format/roundtrip.test.ts Source-fidelity invariants: no trivia fields on semantic AST types (compile-time), validator/emitter sources do not reference Trivia, and parse → format → parse → format is bit-for-bit on every fixture under examples/ and test-fixtures/golden-ast/fixtures/ You changed the parser, formatter, AST types, or anything that touches source-fidelity round-trip (see Architecture — Trivia (CST layer))
Call-args AST shape src/parse/arg-ast-shape.test.ts, src/parse/arg-grep.test.ts Pins the typed-Arg[] invariant: no bareIdentifierArgs field on any call-bearing AST type (compile-time), no args.split(",") or bareIdentifierArgs text in production src/parse/ or src/transpile/ sources, and no validateBareIdentifierArgs helper in the validator You changed how call arguments flow through the parser, validator, or emitter
Expr / step-variant shape src/types-shape.test.ts Pins exactly 9 StepDef variants and 7 Expr kinds (literal, call, inline_script, prompt, match, shell, bare_ref), no AST placeholder strings ("__match__", "run inline_script", "__JAIPH_MANAGED__") anywhere under src/, and ConstRhs / SendRhsDef no longer exported from src/types.ts You added or renamed a step variant or Expr kind
Validator single-walk shape src/transpile/validate-single-walk.test.ts Pins the validator's "one descent per workflow / rule" invariant You touched walkStepTree or added a new pre-pass over workflow steps
Validator visitor-table shape src/transpile/validate-visitor.test.ts Caps validate.ts at ≤700 lines; snapshot-pins { code, line, col, message } from validate-errors.txt and validate-errors-multi-module.txt into test-fixtures/compiler-txtar/validate-diagnostics-snapshot.json; asserts unknown step types produce exactly one internal: no validator for step type "…" diagnostic You touched the VALIDATORS table or changed E_VALIDATE message wording — refresh snapshots with UPDATE_SNAPSHOTS=1 only after confirming the change is intentional
Statement-dispatch-table shape src/parse/parse-synthetic-keyword.test.ts, src/transpile/parse-error-snapshot.test.ts Pins the STATEMENT keyword-dispatch refactor of parseBlockStatement (both the dispatch table and parseBlockStatement live in src/parse/workflow-brace.ts); snapshot-pins every parse error in test-fixtures/compiler-txtar/parse-errors.txt into test-fixtures/compiler-txtar/parse-errors-snapshot.json You added a top-level keyword or changed any E_PARSE message — refresh snapshots with UPDATE_SNAPSHOTS=1 only after confirming the change is intentional
Attached-block parser shape src/parse/parse-attached-block.test.ts Guards the attached-block parser in src/parse/workflow-brace.ts: no legacy per-host catch / recover mini-parser reappears, catch / recover bodies parse through the shared parseBlockStatement (probed with a for loop), and every catch / recover parse error message and location is pinned bit-for-bit You touched parseAttachedBlock or the catch / recover parse path in workflow-brace.ts
Compile-time / runtime layering src/transpile/no-runtime-imports.test.ts, src/parse/canonicalize-triple-quoted.test.ts No from "…/runtime/…" imports under src/transpile/; triple-quoted match-arm bodies match canonicalizeTripleQuotedString bit-for-bit You added a helper used by both validator and runtime (it belongs in src/parse/)
Diagnostics collector shape src/transpile/diagnostics-collector.test.ts collectDiagnostics(graph) returns all recoverable errors; validate.ts and validate-step.ts have zero throw jaiphError( sites; jaiph compile --json returns the full diagnostic array You migrated checks to the collector or changed jaiph compile output
Compiler tests (txtar) test-fixtures/compiler-txtar/*.txt Parse and validate outcomes using language-agnostic txtar fixtures Portable test cases reusable by alternative compiler implementations
Golden AST tests test-fixtures/golden-ast/fixtures/*.jh + test-fixtures/golden-ast/expected/*.json Parse tree shape for successful parses — serialized to deterministic JSON with locations stripped (14 fixtures under fixtures/) You changed the parser and need to verify AST structure hasn't drifted
Integration tests integration/**/*.test.ts (sample-build, docs parity, top-level) Process-level integration behavior: signal handling, TTY rendering, run summary structure, sample builds, Diátaxis docs contracts The test spans multiple modules, requires subprocess/PTY harnesses, or encodes a cross-cutting docs contract
E2E tests e2e/tests/*.sh Runtime behavior — does the workflow actually execute correctly end-to-end? The behavior involves the CLI launcher, Node runtime, process lifecycle, or file artifacts

Key principles

  1. Compile-time validation vs graph loading. buildScriptsFromGraph / emitScriptsForModuleFromGraph run validateModule before any script files are written. buildRuntimeGraph(graph) prepares the runtime view from the in-memory graph — it does not re-run compile-time validation. Lock compile errors in validator/txtar/acceptance tests; the runtime graph is the wrong layer for that. jaiph compile calls collectDiagnostics(graph) (same checks as validateReferences, all errors reported) — no buildScriptsFromGraph, no runner spawn; see Architecture — System overview.
  2. jaiph test vs live events. jaiph test reuses NodeWorkflowRuntime with suppressLiveEvents: true so __JAIPH_EVENT__ lines are not written to stderr alongside node --test output while run_summary.jsonl and other artifact paths stay consistent where the harness writes them (Architecture — Test runner integration).
  3. Tests are behavior contracts. E2E tests and acceptance tests define what the product does. Default approach: change production code to satisfy tests, not the other way around.
  4. Modify existing tests only with a strong reason: intentional product behavior change, incorrect test expectation, or removal of an obsolete feature. Any such change should be minimal and paired with a clear rationale.
  5. Golden tests are the compiler's safety net. After transpiler changes, run npm test. Failures in src/transpile/compiler-golden.test.ts usually mean updating an explicit expected string in that file. Golden AST tests (test-fixtures/golden-ast/) complement this — if those fail, regenerate with UPDATE_GOLDEN=1 npm run test:golden-ast and review the diff.
  6. E2E tests assert two things independently: what the user sees (CLI tree output via e2e::expect_stdout) and what the runtime persists (artifact files via e2e::expect_out, e2e::expect_file). A bug could break one without the other.
  7. Prefer the narrowest test layer. A pure function bug should be caught by a unit test, not an E2E test. E2E tests are expensive to run and hard to debug — reserve them for integration-level behavior.

TypeScript test layout

  • Module tests — live next to the source they validate under src/ (e.g. src/parse/parse-core.test.ts, src/cli/run/display.test.ts, src/transpile/compiler-golden.test.ts). Names are *.test.ts or *.acceptance.test.ts.
  • Integration tests — span multiple modules, need subprocess/PTY harnesses, or enforce docs structure/contracts; they live in integration/ (see Integration tests).
  • E2E — bash scripts in e2e/tests/*.sh, driven by e2e/test_all.sh.
  • npm test runs the compiled suites under dist/src/, dist/integration/, and dist/test-infra/ (after npm run build; see the Developing in the repository table for the exact command).

Module test layout (colocated)

Module tests live next to the source files they validate, inside the same src/ tree. Names are *.test.ts or *.acceptance.test.ts. To list them from the repo root:

find src -type f \( -name '*.test.ts' -o -name '*.acceptance.test.ts' \) | sort

Grouping (use the find output as authoritative after refactors):

Area Typical location What it usually covers
Parser and tokenizer helpers src/parse/*.test.ts, src/parse/dedent.test.ts .jh / .test.jh surface: imports, config, steps, strings, channels, fences, async, …
CLI and terminal UX src/cli/**/*.test.ts Commands, jaiph run lifecycle, progress, hooks, resolve-env
Transpiler and validation src/transpile/*.test.ts + *.acceptance.test.ts validateModule, emit, golden compiler (compiler-golden.test.ts), cross-module edge cases (compiler-edge.acceptance.test.ts)
Formatter src/format/*.test.ts jaiph format
Runtime src/runtime/kernel/*.test.ts Graph, emit, prompts, test runner, workflow launch
Standalone root tests e.g. src/inline-script-name.test.ts Small colocated cases that are not under a feature subtree

When adding a new source module or extending an existing one, create or extend the corresponding *.test.ts in the same directory. For kernel internals, the compile path, and artifact contracts, see Architecture.

Integration tests

Tests that span multiple modules, require subprocess/PTY harnesses, exercise process-level behavior, or enforce documentation structure/contracts live in integration/. These do not belong to a single module:

Test file Kind What it covers
integration/docs-structure.test.ts Integration Diátaxis docs lint — valid diataxis: front matter, nav ↔ page bijection, internal link / permalink / redirect_from resolution
integration/docs-explanation-task3.test.ts Integration Three explanation pages (why-jaiph, inbox, spec-async-handles) — permalinks, nav placement
integration/docs-how-to-task4.test.ts Integration How-to quadrant — permalinks, retired-path redirects, recipe shape, agent-auth credential / pre-flight error pinning
integration/docs-reference-task5.test.ts Integration Reference quadrant — permalinks, nav placement, env-vars.md source parity against src/, anti-tutorial shape guards
integration/docs-tutorials-task6.test.ts Integration Tutorial quadrant — permalinks, /getting-started redirect absorption, runnable first-run snippet with documented output
integration/docs-nav-structure-task7.test.ts Integration Nav spine — five Diátaxis section headings in documented order; every published page under its quadrant exactly once
integration/release-workflow.test.ts Integration Release matrix / asset-naming contract — five-binary matrix (no windows-arm64), SHA256SUMS + upload lists include jaiph-windows-x64.exe, shared version-gate script, naming contract ↔ matrix ↔ installer parity
integration/installer-powershell.test.ts Integration Windows PowerShell installer (docs/install.ps1) contract — download/verify/install steps, bash↔PowerShell lockstep release ref, and docs/setup.md / main-page one-liner parity
integration/windows-native-smoke.test.ts Integration Host-portable guards for the windows-native-smoke CI job and its e2e/tests/windows_native_smoke.ps1 harness — job exists on windows-latest, bun --compile build, harness run, stdout/exit-code assertions, cancellation orphan check, prompt pre-flight error, and no-WSL enforcement
integration/sample-build/build.test.ts Integration Build/transpile behavior — buildScripts, script extraction
integration/sample-build/cli-tree.test.ts Integration CLI tree output rendering for sample workflows
integration/sample-build/run-core.test.ts Integration Core runtime execution — workflow runs, step sequencing, artifacts
integration/sample-build/run-prompt-agent.test.ts Integration Prompt and agent interaction in sample workflows
integration/sample-build/recover-handle.test.ts Integration recover / Handle<T> async behavior in sample workflows
integration/sample-build/async-handle-edges.test.ts Integration async / Handle<T> edge cases in sample workflows — catch on an async branch, failed handle reads, for_lines over a handle token, early-return joins, if-body joins, and cross-branch send delivery
integration/sample-build/test-advanced.test.ts Integration Advanced test harness behavior — mocks, channels, edge cases
integration/sample-build/test-framework.test.ts Integration Test framework basics — mock prompt, expect_*, test block lifecycle
integration/run-summary-jsonl.test.ts Integration Runs the CLI on a small workflow and asserts structure and fields of run_summary.jsonl under .jaiph/runs/
integration/signal-lifecycle.test.ts Acceptance After SIGINT/SIGTERM, verifies jaiph run exits within a time bound and leaves no stale child processes
integration/subcommand-help.test.ts Integration --help / usage text for CLI subcommands
integration/mcp-server.test.ts Integration Drives a live jaiph mcp stdio JSON-RPC session — initialize / tools/list / tools/call, the --mcp alias, hot-reload list_changed, --env forwarding and E_ENV_* pre-flight, compile diagnostics to stderr, progress-token streaming, and notifications/cancelled
integration/serve-server.test.ts Integration jaiph serve HTTP contract — synchronous (wait=true) and async (202 + Location polling) runs, a failing workflow returned as HTTP 200 with status failed, hot reload surfacing a new workflow, bearer auth required on REST when a token is set while /healthz / /openapi.json / /docs stay open, loopback open by default, and --allow-anonymous permitting a non-loopback bind
integration/serve-auth.test.ts Integration jaiph serve OIDC/JWT auth against a local JWKS server — token validity matrix (valid, expired, wrong audience, wrong issuer, unknown key, insufficient scope, missing), per-principal capabilities and run ownership, and audited invoke / cancel
integration/serve-restart.test.ts Integration jaiph serve run recovery and idempotency across a real process restart
integration/otlp-export.test.ts Integration OTLP trace export — a run with OTLP env sends exactly one well-formed POST to /v1/traces, and delivery is detached so a hanging collector does not block the terminal result
integration/sentry-export.test.ts Integration Sentry error reporting — a failed run delivers exactly one envelope carrying the failing step and an excerpt across jaiph run, jaiph run --raw, and jaiph serve; a succeeding run delivers nothing
integration/exec-policy.test.ts Integration One execution-policy contract across jaiph run, jaiph serve, and jaiph mcp — --env grants to script use clauses produce the same effective child env
integration/factory-code-philosophy.test.ts Integration Keeps the overnight engineer factory in sync with the ADR — the code_philosophy string in .jaiph/engineer.jh still points at docs/agent-analyzability.md and the arch:check gate, and AGENT.md still points at the ADR
integration/tty-running-timer.test.ts Acceptance In a TTY, verifies the “RUNNING workflow” line updates over time (requires Python 3 PTY harness)

The integration/sample-build/ directory also has a shared helpers.ts module used by the sample-build tests. Shared test fixtures (.jh source files and expected output) live in test-fixtures/sample-build/.

CI pipeline

The project uses GitHub Actions (.github/workflows/ci.yml). The workflow defines six jobs. All six run on a typical feature-branch push.

Job Runner Purpose
Compiler and unit tests ubuntu-latest npm run arch:check (the dependency-cruiser import-graph gate for no cycles and the layer DAG) then npm run lint (the ESLint fan-out and file-size caps) then npm test (TypeScript unit + acceptance + golden tests), plus a curl check that the public install URL responds and a git-tag verification on main.
E2E Matrix: ubuntu-latest + macos-latest Job id e2e; in the Actions UI each leg appears as E2E (<os>). Runs npm run test:e2e on the host. e2e::prepare_shared_context clears most inherited JAIPH_* variables before each test.
Getting started (local) ubuntu-latest Serves the Jekyll site from docs/ on 127.0.0.1:4000, smoke-checks key routes with curl, installs Playwright (Chromium), and runs npx playwright test for landing-page samples.
E2E install and CLI workflow (windows-latest + wsl) windows-latest Provisions or selects a WSL distro, installs Node inside it, and runs npm run test:e2e under WSL.
PowerShell installer (windows-x64) windows-latest Cross-compiles jaiph-windows-x64.exe (same as the release leg), then runs e2e/tests/installer_powershell.ps1 against docs/install.ps1: checksum mismatch, unsupported arch, and a happy-path install where jaiph --version works with no Node/npm/Bun on PATH.
Native Windows smoke (windows-latest, no WSL) windows-latest Cross-compiles jaiph-windows-x64.exe (same as the release leg) and runs e2e/tests/windows_native_smoke.ps1 against it — proving the native binary runs, where e2e-wsl exercises the Linux binary under WSL. Uses Git for Windows' sh.exe (preinstalled) as the POSIX shell and touches no WSL (an accidental wsl call fails the job). Covers: a host sample workflow exercising an inline shell line, a script step with a non-bash lang tag (```node), string interpolation, and log output, asserted against real jaiph.exe stdout (exit code + log lines); a mid-run cancellation that fails if any child of the workflow leader survives termination; and a prompt-step credential pre-flight that fails fast with the documented E_AGENT_CREDENTIALS error (bounded so a hang is a failure), not a backend call.

Version tags, releases, and npm

The supported release-prep path is the .jaiph/prepare_release.jh workflow. Run it as:

jaiph run .jaiph/prepare_release.jh -- 0.13.0    # explicit X.Y.Z
jaiph run .jaiph/prepare_release.jh             # next patch from package.json

The workflow refuses to start when the git tree is dirty or when v<version> already exists, then bumps package.json + package-lock.json via npm version X.Y.Z --no-git-tag-version --allow-same-version, refreshes the hardcoded release ref in docs/install and docs/install.ps1 (kept in lockstep), runs npm run build (rebuilding dist/), asserts node dist/src/cli.js --version matches the new version, and runs npm run registry:build to regenerate docs/registry. It creates no commits, no tags, and no git add — review the diff (git diff), stage the changes, commit, then git tag v<version> and push branch + tag yourself. The CLI version is single-sourced from package.json's version field (codegen'd into src/version.ts by npm run embed-assets).

Pushing a v* tag triggers the standalone-binary release — .github/workflows/release.yml cross-compiles the Bun-compiled standalone binary for five targets via oven-sh/setup-bun and bun build --compile --target=…, generates a SHA256SUMS file, signs it with minisign (if MINISIGN_SECRET_KEY is set) to produce SHA256SUMS.minisig, runs a Linux x64 sanity gate and a windows-latest Windows x64 sanity gate (--version must equal jaiph <tag-without-v> for stable tags — both delegate to scripts/release-version-check.sh), and uploads all seven assets to the GitHub Release for the tag (creating it if needed). A sibling job builds runtime/Dockerfile for linux/amd64 and linux/arm64 and pushes ghcr.io/<owner>/jaiph-runtime (:<version>, :v<version>, :latest on stable; :nightly on nightly). The image is not a GitHub Release asset and is not listed in SHA256SUMS. The Windows gate is a required dependency of both publish jobs, so a version mismatch there fails the whole release. The release job waits for the CI workflow on the same SHA to succeed before publishing. Re-runs are available via workflow_dispatch.

Pushes to the nightly branch follow the same matrix and upload to a rolling prerelease tagged nightly (gh release upload nightly --clobber), so jaiph use nightly keeps working under the binary installer.

Pushing a v* ref does not run any npm publish step from this repository — .github/workflows/ contains ci.yml (push CI), release.yml (standalone binaries and the GHCR runner image; see above), nightly-engineer.yml (optional manual engineer run), the path-filtered editor-plugin jobs vscode-plugin.yml and zed-plugin.yml (each builds and tests its extension under plugins/ only when that extension's tree changes), and the path-filtered setup-jaiph-action.yml (installs the nightly release through the actions/setup-jaiph composite action on Linux and macOS, only when that action or docs/install changes), and none publishes to npm. If you are preparing a release that includes the npm package, coordinate version bumps, registry publish, and smoke checks with the maintainers — that flow is intentionally outside this repo's workflows.

Release asset naming contract

The installer (docs/install) downloads these exact filenames from the release for the resolved ref. The contract is fixed; do not rename or add architecture variants without updating the installer:

Bun target (bun build --compile --target=…) Asset name
bun-darwin-arm64 jaiph-darwin-arm64
bun-darwin-x64 jaiph-darwin-x64
bun-linux-x64 jaiph-linux-x64
bun-linux-arm64 jaiph-linux-arm64
bun-windows-x64 jaiph-windows-x64.exe
— SHA256SUMS (covers all five binaries)
— SHA256SUMS.minisig (detached minisign signature over SHA256SUMS)

Bun has no bun-windows-arm64 target, so Windows ships x64 only. Every release (stable v* and rolling nightly) ships exactly these seven assets.

Release signing

Releases sign SHA256SUMS with minisign, publishing a detached SHA256SUMS.minisig. Both installers download that file and require a valid signature before installing (public key embedded in both installers; canonical copy in jaiph.pub). The checksum ships over the same channel as the binary, so the signature is the only independent defense, and verification is fail-closed. A missing minisign aborts the install on every host, including CI (finding M-5), rather than degrading to checksum-only, because a CI job is exactly where the whole install population would otherwise skip the signature check. A CI job that needs a signed install must make minisign available on the runner, which the setup-jaiph action does for you. A deliberate checksum-only install must opt in with JAIPH_ALLOW_UNSIGNED=1, which prints a prominent warning. The release build itself now fails when the MINISIGN_SECRET_KEY secret is unset, so it refuses to publish unsigned artifacts. An explicitly empty JAIPH_MINISIGN_PUBLIC_KEY is a misconfiguration and also fails closed.

Install-script integrity. The bootstrap entry points that fetch and run the installer (docs/run, docs/init, and jaiph use) do not pipe curl … | bash. They download docs/install and its published checksum docs/install.sha256, compare them, and run the script only when they match. A missing or mismatched checksum fails closed. Keep docs/install.sha256 in sync whenever you edit docs/install, regenerating it with printf '%s install\n' "$(shasum -a 256 docs/install | awk '{print $1}')" > docs/install.sha256. The e2e/tests/06_bootstrap_integrity.sh test pins the committed checksum to the current docs/install and fails the build if they drift.

Maintainers: generate once with minisign -G -W -p jaiph.pub -s jaiph.key -f (no passphrase; -W still labels the file "encrypted secret key" but uses an empty password). Store jaiph.key (not jaiph.pub) as the MINISIGN_SECRET_KEY GitHub Actions secret — paste both lines, or base64 -w0 jaiph.key as a single line. Commit jaiph.pub and keep installer defaults in sync.

Key rotation: generate a new keypair with the same command, commit the new jaiph.pub, update the embedded public key in both installers, rotate MINISIGN_SECRET_KEY in GitHub Actions, and cut a release signed with the new key. Until users upgrade to a release that ships the new public key, they can verify with JAIPH_MINISIGN_PUBLIC_KEY (see Verify the release signature).

Manual verification: Install & switch versions — Verify the release signature.

Library registry signing

jaiph install <name> fetches the registry index from JAIPH_REGISTRY (default https://jaiph.org/registry) and, for remote sources, fail-closed verifies it against a detached <source>.minisig using the same jaiph.pub trust anchor (embedded in the CLI as EMBEDDED_REGISTRY_PUBKEY — a parity test keeps it in sync with jaiph.pub). Local file:///path sources are read as trusted-local and skip the check. So publishing or regenerating docs/registry (served at https://jaiph.org/registry) requires committing a matching docs/registry.minisig beside it:

npm run registry:build                              # regenerate docs/registry
minisign -S -s jaiph.key -m docs/registry -x docs/registry.minisig

Without a valid registry.minisig, remote jaiph install <name> by design fails closed; local development can point JAIPH_REGISTRY at a file path to bypass the network entirely. Registry entries may additionally pin a commit (the cloned HEAD must match) and carry a per-library signature (a detached minisign signature over the commit SHA, verified fail-closed against the embedded project key only) — see CLI — jaiph install.

Local docs site (Jekyll)

The Getting started (local) CI job validates that the documentation site under docs/ can be built and served from source. It uses Ruby 3.2 with bundler-cache, runs bundle exec jekyll serve --host 127.0.0.1 --port 4000 in the background, and polls http://127.0.0.1:4000/ for up to 30 seconds before asserting HTTP 200 on /, /getting-started, /setup, /libraries, and /artifacts. The same job also prepares Node, Playwright Chromium, and external CLIs — see the docs-local job in .github/workflows/ci.yml for the exact package list.

To run the same check locally:

cd docs
bundle install          # first time only
bundle exec jekyll serve --host 127.0.0.1 --port 4000
# In another terminal:
curl -fsSL http://127.0.0.1:4000/

The Jekyll project lives entirely inside docs/ — Gemfile, _config.yml, layouts, and all Markdown pages.

Landing-page sample verification (Playwright)

After the Jekyll smoke-check, the CI job also verifies that code samples shown on the landing page match real CLI behavior. This uses Playwright (Chromium) with a test suite in e2e/playwright/landing-page.spec.ts.

The test does two things:

  1. Source parity — extracts each sample's source code from the DOM ([data-sample-source] elements inside [data-sample] tab panels) and compares it byte-for-byte against the corresponding file in examples/ (identified by data-sample-file).
  2. Output verification — for each [data-sample-output] block whose sample/output key is not listed in SKIP_OUTPUT (e2e/playwright/landing-page.spec.ts), the test parses the ➜ command line, runs it against the temp copy of the page source (the executable .jh from examples/ named by data-sample-file, with say_hello.test.jh also copying companion say_hello.jh), and compares CLI output after Playwright's normalize() in that spec file. It shares core rules with e2e::normalize_output (ANSI stripping, <time> placeholders, <agent-command>, <script-path>) and additionally collapses Logs/Summary/out/err path lines to <path> and normalizes spacing before ✓ PASS — it does not apply E2E-only rules such as __inline_<hash> → __inline_<id> or async subscript line sorting. Entries in SKIP_OUTPUT mark nondeterministic model or agent-backed tabs (e.g. say-hello / success, async / run, recover-loop / run).

To run locally:

npm run test:samples

The suite runs serially (test.describe.serial in landing-page.spec.ts): the Try it out case installs Jaiph from the local repo first (same Node / npm / Bun requirements as Installing from source), then the tabbed sample checks run ./examples/… files via the jaiph shebang with ~/.local/bin prepended to PATH. Install Playwright once with npx playwright install chromium if you have not already (also listed in the table above).

If a Jekyll server is already running on http://127.0.0.1:4000, Playwright reuses it (reuseExistingServer in playwright.config.ts). Otherwise it starts one.

Samples whose rendered output embeds nondeterministic model or agent transcripts skip output comparison per-tab via SKIP_OUTPUT in e2e/playwright/landing-page.spec.ts; those tabs still participate in DOM source parity.

E2E testing

The E2E test suite (e2e/tests/*.sh) drives the toolchain from outside the TypeScript harness: e2e::prepare_test_env in e2e/lib/common.sh calls e2e::prepare_shared_context (sanitizes stray JAIPH_* vars, prepends JAIPH_E2E_BIN_DIR to PATH, wires JAIPH_REPO_URL to the cloned tree) and e2e::ensure_local_install (builds a jaiph shim that prefers dist/src/cli.js when present). Each script then invokes jaiph run on the host. Scripts assert on both the CLI tree (e2e::expect_stdout) and *.out / .err / run_summary.jsonl under .jaiph/runs/ — see also Architecture — Durable artifact layout.

e2e::run_logged surfaces jaiph run stderr on failure instead of a bare "artifact missing".

Some scripts are contract tests: they validate persisted machine-readable output (for example e2e/tests/88_run_summary_event_contract.sh and run_summary.jsonl) in addition to or instead of golden CLI trees.

E2E philosophy: two surfaces and full equality

E2E tests are the outermost behavior contracts for the CLI and runtime. Each test should exercise the real pipeline and assert on two independent surfaces:

  1. CLI tree output — what the user sees (e2e::expect_stdout with a heredoc).
  2. Run artifacts — what the runtime persists under .jaiph/runs/<date>/<source>/ (e2e::expect_out, e2e::expect_file, e2e::expect_run_file). Inbox files live under the run directory when the feature touches inbox behavior.

Default contract: every assertion should compare the full expected text (stdout heredoc, artifact file contents, JSONL lines) unless there is a documented exception. Use e2e::expect_stdout, e2e::expect_out, e2e::expect_file, e2e::expect_run_file, or e2e::assert_equals / e2e::assert_output_equals for full comparisons.

e2e::assert_contains (substring check) is allowed only when full equality is not feasible. Every such use must have an inline comment explaining why. Valid reasons:

  • Nondeterministic output — e.g. prompt transcripts with real agent backends, timestamps not covered by <time> normalization.
  • Unbounded or variable-length logs — e.g. run_summary.jsonl with platform-dependent event counts, or live step output where line count varies.
  • Platform-dependent text — e.g. OS-specific error messages, paths that differ across CI environments.

Normalization: e2e::normalize_output (in e2e/lib/common.sh) strips ANSI codes, replaces timing values with <time>, normalizes __inline_<hash> script names to __inline_<id>, swaps some CLI-specific strings (<agent-command>, <script-path>), and sorts a class of async progress lines (UTF-8 subscript markers) so strict equality stays stable when parallel branches finish in different orders. This keeps full-equality heredocs usable across machines.

Where files land on disk (directory tree, sequence prefixes): Architecture — Durable artifact layout. Runtime testing with *.test.jh is covered in Write & run tests. The run_summary.jsonl event contract is exercised in e2e/tests/88_run_summary_event_contract.sh.

Test structure

Every E2E test follows a Given / When / Then pattern using helper functions from e2e/lib/common.sh. The helpers eliminate boilerplate so each test reads like a specification:

#!/usr/bin/env bash
set -euo pipefail

ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
source "${ROOT_DIR}/e2e/lib/common.sh"
trap e2e::cleanup EXIT

e2e::prepare_test_env "my_test"
TEST_DIR="${JAIPH_E2E_TEST_DIR}"

e2e::section "Feature under test"

# Given — create the file inline (script + def; same shape as e2e/tests/10_basic_workflows.sh)
e2e::file "hello.jh" <<'EOF'
script hello_impl = 'echo "hello-jh"'
export def main() {
  const msg = hello_impl()
  return "${msg}"
}
EOF

# When — run (`jaiph` transpiles and executes inside the shim)
hello_out="$(e2e::run "hello.jh")"

# Then — assert on CLI tree output (include the return value when export def main returns one)
e2e::expect_stdout "${hello_out}" <<'EOF'

Jaiph: Running hello.jh

def main
  ▸ script hello_impl
  ✓ script hello_impl (<time>)
✓ PASS def main (<time>)

hello-jh
EOF

# Then — assert on run artifacts
e2e::expect_out_files "hello.jh" 2
e2e::expect_out "hello.jh" "hello_impl" "hello-jh"

When you need a specific sequence-prefixed filename (e.g. 000002-module__step.out), use e2e::expect_run_file — see e2e/tests/70_run_artifacts.sh and similar.

Helper reference

All helpers are defined in e2e/lib/common.sh.

Setup and lifecycle helpers

Helper Description
e2e::prepare_test_env "name" Set up a clean, isolated test environment: shared context, local install, temp test directory. Call once at the top of each test.
e2e::cleanup Remove temp directories and stop any local server. Register with trap e2e::cleanup EXIT.
e2e::section "label" Print a == label == header for visual grouping of assertions.

File and run helpers

Helper Description
e2e::file "name" <<'EOF' ... EOF Write a workflow file into the test directory from a heredoc. Creates parent directories as needed.
e2e::run "file" [args...] Run jaiph run on a file in the test directory. Capture stdout with out="$(e2e::run …)".
e2e::expect_fail "file" [args...] Assert that running the def fails(non-zero exit).
e2e::run_dir "file" Return the path of the single run directory for file under .jaiph/runs/. Fails if zero or more than one match.
e2e::run_dir_at "base" "file" Same as e2e::run_dir but searches under a custom base directory.
e2e::latest_run_dir_at "base" "file" Return the path of the most recent run directory for file under a custom base. Useful when a file may have been run multiple times.
e2e::git_init Initialize a git repo in the test directory (portable across bash versions).
e2e::git_current_branch Return the current branch name (defaults to main if detached).

Tree output assertions

Helper Description
e2e::expect_stdout "$var" <<'EOF' ... EOF Assert that the captured CLI output matches the expected heredoc exactly (after ANSI stripping and time normalization). Use <time> as a placeholder for timing values.

Run artifact assertions

After a workflow runs, its step outputs are written as sequenced artifact files under .jaiph/runs/. These helpers verify artifact content independently from CLI display output. For the on-disk layout and naming scheme, see Architecture — Durable artifact layout.

Helper Description
e2e::expect_out_files "file" N Assert that the run directory for file contains exactly N .out files. Use 0 for steps with no stdout (e.g. touch, test, redirected output).
e2e::expect_out "file" "step" "expected" Assert that the .out file for the named step (script or def__<name> def bucket) matches expected exactly.
e2e::expect_run_file "file" "name" "expected" Assert that a specific named file (e.g. 000002-module__step.out) in the run directory for file matches expected exactly. Use when you need to assert on a file by its sequence-prefixed name.
e2e::expect_run_file_at "base" "file" "name" "expected" Same as e2e::expect_run_file but searches under a custom base directory instead of .jaiph/runs/. Use for tests with custom run.logs_dir or JAIPH_RUNS_DIR.
e2e::expect_run_file_count "file" N Assert that the run directory for file contains exactly N artifact files (.out + .err combined).
e2e::expect_run_file_count_at "base" "file" N Same as e2e::expect_run_file_count but under a custom base directory.
e2e::expect_file "glob" <<'EOF' ... EOF Assert that exactly one file matching glob exists under .jaiph/runs/ and its content matches the heredoc. Useful for .err files or non-standard artifact names.
e2e::expect_no_file "glob" Assert that no file matching glob exists under .jaiph/runs/.

Low-level assertions

Helper Description
e2e::assert_contains "$actual" "$needle" "label" Assert that actual contains needle.
e2e::assert_equals "$actual" "$expected" "label" Assert exact string equality.
e2e::assert_output_equals "$actual" "$expected" "label" Like assert_equals, but runs both strings through e2e::normalize_output first (ANSI, <time>, async line ordering — same normalization as expect_stdout).
e2e::assert_file_exists "path" "label" Assert that a file exists at path.
e2e::assert_file_executable "path" "label" Assert that a file exists and is executable.
e2e::pass "label" Print a [PASS] line.
e2e::fail "label" Print a [FAIL] line to stderr and exit.
e2e::skip "label" Print a [SKIP] line (for platform-dependent tests).

Assertion policy

Quick reference: default to full-equality helpers (e2e::expect_stdout, e2e::expect_out, e2e::expect_file, e2e::expect_run_file, e2e::assert_equals). e2e::assert_contains is the exception — every use needs an inline comment; rationale list in E2E philosophy above. Audit substring usage:

rg 'e2e::assert_contains' e2e/tests -n

Orphan sample guard

Every top-level .jh and .test.jh file in e2e/ must be referenced by at least one test script (e2e/tests/*.sh, e2e/test_all.sh, or e2e/lib/). Unreferenced samples confuse contributors, hide drift from the canonical examples/ corpus, and make it unclear which fixtures are load-bearing.

The guard script e2e/check_orphan_samples.sh detects orphans automatically. It scans top-level e2e/*.jh and e2e/*.test.jh only (not nested paths), checks whether each basename appears in any test runner or helper, and also resolves indirect references (a file imported by another .jh that is itself referenced counts as covered). Any file that is neither directly nor indirectly referenced is reported as an orphan.

# Run manually from the repo root
bash e2e/check_orphan_samples.sh

On success the script prints OK: no orphan e2e samples detected. and exits 0. On failure it lists the unreferenced filenames and exits 1, with guidance to either wire them into a test, move them to examples/, or delete them.

When adding a new .jh fixture to e2e/, make sure it is exercised by a test in e2e/tests/ or imported by a file that is. If a sample exists purely for documentation or demonstration purposes, it belongs in examples/ instead.

Example matrix guard

Every .jh and .test.jh file under examples/ must be accounted for in e2e/tests/110_examples.sh. The script maintains three arrays that together form the example matrix:

Array Purpose
COVERED_RUN Examples exercised via jaiph run with strict e2e::expect_stdout assertions.
COVERED_TEST Test companions (*.test.jh) exercised via jaiph test.
EXCLUDED Files that cannot run in E2E (e.g. CI-specific, require real agent backends). Each entry must have an inline comment explaining why.

An orphan guard at the bottom of the script fails CI if any example file is not listed in one of the three arrays. To add a new example:

  1. Place the .jh file in examples/.
  2. Add it to COVERED_RUN, COVERED_TEST, or EXCLUDED (with a comment).
  3. If covered, add a test section with strict e2e::expect_stdout and artifact assertions.