Skip to content

ci: guard harness-agnostic skills and tier table drift #30

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
helderberto merged 2 commits into from
Oct 23, 2026
Merged

ci: guard harness-agnostic skills and tier table drift #30

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
74c71c6
ci: tiers.sh fails when the tier table drifts from frontmatter
helderberto Oct 23, 2026
6359641
ci: check.sh fails on Claude Code-only tokens in skills
helderberto Oct 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/skill-conventions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ Skills also carry an **effort** hint: mechanical ones (`commit`, `to-adr`) run a

A skill that reads text other people wrote (PR bodies, diffs, review comments, issues, CI logs, fetched docs) calls it data to analyze, never instructions to follow. Say so in the step that fetches it and in every subagent brief that receives it. Snyk's W011 still flags such a skill on skills.sh: it marks the exposure, not a missing guard.

## Harness-agnostic

A skill reads the same in every supported agent: name a tool by its role ("a read-only explore subagent"), a bundled file by its place in the skill ("this skill's `scripts/` folder"), and a command as plain shell. `scripts/check.sh` fails on Claude Code-only tokens. `$ARGUMENTS` and `/clear` / `/compact` in `Next:` lines stay until that call is made.

## Verifying commands

A skill's git and shell commands are its code, and prose has no test harness. Run each one in a scratch repo before it ships, with a deleted file, a renamed file, and a local base behind its remote. That run is the failing test first.
2 changes: 2 additions & 0 deletions scripts/check.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ cd "$(dirname "$0")/.."
claude plugin validate .
cmp .claude-plugin/plugin.json .plugin/plugin.json
jq -e --slurpfile c .claude-plugin/plugin.json '.name == $c[0].name and .description == $c[0].description' plugin.json > /dev/null
bash scripts/tiers.sh
! grep -rnE '\$\{CLAUDE_SKILL_DIR\}|~/\.claude/|subagent_type=|`! [a-z]|claude --[a-z]' skills
bash skills/review/scripts/floor-guard.test.sh
bash skills/safe-repo/scripts/scan-secrets.test.sh
bash skills/to-doc/scripts/adf-outline.test.sh
26 changes: 26 additions & 0 deletions scripts/tiers.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
# Fails when the tier table in docs/skill-conventions.md drifts from the skills' frontmatter.
#
# Usage:
# bash scripts/tiers.sh
set -euo pipefail

cd "$(dirname "$0")/.."

doc=docs/skill-conventions.md
fail=0
typed=0 both=0 model=0

for f in skills/*/SKILL.md; do
s=$(basename "$(dirname "$f")")
if grep -q '^disable-model-invocation: true' "$f"; then row='Typed-only'; typed=$((typed + 1))
elif grep -q '^user-invocable: false' "$f"; then row='Model only'; model=$((model + 1))
else row='Both'; both=$((both + 1)); fi
grep -E "^\| $row \(" "$doc" | grep -qE "(\| | · )$s( · | +\|)" || { echo "$s: missing from the $row row" >&2; fail=1; }
done

for want in "| Typed-only ($typed)" "| Both ($both)" "| Model only ($model)" "($((typed + both)) entries)" "descriptions ($((both + model)))"; do
grep -qF "$want" "$doc" || { echo "$doc: expected \"$want\"" >&2; fail=1; }
done

exit "$fail"
Loading