Skip to content

noun: fix validation, refcounts, and results in lagoon jets - #1057

Open
sigilante wants to merge 1 commit into
developfrom
sigilante/lagoon-check-fixes
Open

noun: fix validation, refcounts, and results in lagoon jets#1057
sigilante wants to merge 1 commit into
developfrom
sigilante/lagoon-check-fixes

Conversation

@sigilante

@sigilante sigilante commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Prompted by ~dozreg-toplud's refcount linter, which flagged leaked nouns in the lagoon jets' _check, plus a hand audit of the surrounding code against the Hoon library the jets serve. The audit turned up considerably more than the linter findings: broken validation, jet/Nock semantic divergences, wrong-result kernel bugs, and a family of leaks. Everything lands in pkg/noun/jets/i/lagoon.c.

_check helper

The old _check was wrong in four independent ways:

  • u3qa_dec's result was never freed (the linter's finding), and u3r_met's raw c3_w return was type-punned through u3_atom.
  • The shape product was computed in c3_d, truncated into a u3_atom, and compared word-for-word against a noun — so any well-typed ray with ≥ 2³¹ elements spuriously failed _check (jet crashes where the Nock succeeds), and a crafted shape whose product wraps could pass it.
  • The product could silently overflow 64 bits.
  • The bloq went into u3r_met unvalidated; u3r_met requires a_y < 37 and only enforces it with c3_dessert, which release builds compile out — undefined behavior on a hostile bloq.

The rewrite takes meta/data as two borrowed nouns (no cell is consed, nothing is allocated), computes the product with overflow detection (an overflowed product cannot equal the block count of any real atom, which mirrors the Hoon), handles bloqs ≥ 37 the way +met does, and bails %exit only where +check itself crashes (cell dims, improper shape list, zero data underflowing +dec).

Every call site previously passed _check(u3nc(x_meta, x_data)). u3nc transfers ownership, but the metas/datas are borrowed from the sample, and the cell was never freed — one leaked cell per jet invocation, with the ownership violation masked by the leak (naively freeing the cell would have corrupted the sample's refcounts).

Wrapper ↔ Hoon divergences

  • transpose/diag passed the whole core to _check (_check(cor)), which then read the gate's battery as ray metadata — these two jets unconditionally bailed %exit whenever they fired. This had gone unnoticed because there are no transpose tests in the desks that exercise the jets.
  • The elementwise (+add/+sub/+mul/+div/+mod), comparison (+gth/+gte/+lth/+lte), scalar (+add-scalar &c), +abs, and +trace wrappers never called _check at all, though the Hoon asserts it — the jet computed a result where the Nock deterministically crashes.
  • +ravel has no +check and never reads the kind, so its wrapper now punts (not bails) on an inconsistent ray.
  • +dot only asserts equal shapes outside the %i754 path; the full meta-equality and consistency assertions moved inside the %i754 case so other kinds punt to the Nock instead of bailing.
  • +mmul never compared the two rays' bloq/kind (the Hoon computes elementwise on each ray's own meta; the kernel assumes one width), and its gemm switch had no default — an out-of-range bloq silently returned a zero matrix. Both cases now punt.
  • Reduction results (+cumsum/+min/+max/+dot) hard-coded shapes ~[1], ~[1 1], or ~[n 1]; +scalar-to-ray actually returns an all-ones shape of the input's rank. The dot wrapper's ~[n 1] produced a shape/data-inconsistent ray outright.
  • _set_rounding now reports an unrecognized rounding mode so callers punt, instead of u3m_bail(c3__fail).

Kernel bugs (wrong results)

All previously unobservable because the wrapper bailed before reaching them, or untested:

  • trace computed (dot d d) — the sum of squares of the diagonal — instead of (cumsum (diag a)). Identity-matrix tests mask this (1² = 1).
  • transpose read and wrote with rows/columns swapped (correct only for square matrices), and the wrapper returned the unswapped shape.
  • argmin/argmax returned len - i - 1 instead of the ravel index i.
  • diag and abs return elements in natural order, in lockstep with lagoon: fix element order in +diag/+abs, result shape in bloq-4 +range urbit#7388, which removes erroneous flops in +diag and +el-wise-op (the "compensate for LSB" comment there is mistaken — +rep puts the first list element lowest, so the flop reverses the elements). Skew note: the old abs jet already returned natural order, so jetted and unjetted ships disagree about +abs today — the Hoon fix converges with deployed jet behavior; the old diag jet never returned at all. The two PRs should land together.
  • trace honors the door rounding mode (its sum previously used whatever softfloat_roundingMode was left over); linspace fences an indirect count atom before using it as a raw integer in allocation sizes.
  • mod (ray-ray and scalar) rounded the quotient with a hardcoded round-toward-zero, but +toi rounds in the door mode — 7 mod 2 is -1 under %n, not 1. It also returned values where the Hoon crashes: a non-finite quotient (zero or NaN divisor) is (need ~). And mod-scalar/div-scalar multiplied by a rounded 1/n instead of dividing — wrong even for exact quotients (21 mod 7 gave 7; 21/7 gave 2.9999998). All four kernels now divide directly, round via softfloat_roundingMode, and bail %exit on a non-finite quotient.
  • gth/gte used SoftBLAS's f_gt/f_ge macros, which are (!le)/(!lt)true whenever either operand is NaN, where IEEE gt/ge are false. The same disease reaches min/max/argmax through f_min/f_max. All now use lt/le with swapped operands, which also reproduces the Hoon reel-fold exactly (NaN at the head is sticky, interior NaN is skipped, first-of-ties wins).
  • range counted elements with a one-shot ceil((b-a)/d), but the Hoon iterates x+d in the door mode testing each sum against b — for d = .0.1 the two shapes disagreed outright (10 vs 11 elements). The kernel now replicates the iteration (the values are the accumulated sums), the wrapper takes the count from the result's block count, and it punts on non-finite bounds/step, a zero step, or a stalled accumulator (x+d == x short of b), where the Nock loops forever.

Also fixed leaks of _get_dims arrays in the diag/dot wrappers and trace kernel (with raw c3_d dimensions used directly as nouns — invalid for dims ≥ 2³¹), the argmin/argmax scan buffers, and over-kept result shape cells in linspace/range.

Verification

Fresh fake ship (-F bus, urbit-408k pill) on this branch, against the base lib/lagoon.hoon with the urbit/urbit#7388 flop fixes applied (its %add-rays &c hints match the tree registration), differentially compared with a de-jetted copy of the same library (parent ~% %non hint renamed so nothing registers):

  • 36/36 elementwise-equality checks pass, including: transpose/mmul of non-square matrices, diag/trace, reduction shapes at rank 1, argmin/argmax with extrema away from index 0 and with NaN at and after the head, mod under %n/%u, mod-scalar and div-scalar on exact quotients, gth/gte/lth against NaN, abs on negative entries, ravel, linspace, range with d = .0.1 and with a negative step, and add/cumsum/mod-scalar/range under non-default rounding modes via +lake.
  • Zero-divisor mod crashes on both doors (jet bails %exit where the Nock's (need ~) crashes).
  • Jets confirmed firing, not punting: 40×40 %i754 mmul+trace runs in ≤3 s jetted vs ~25 s interpreted, with bit-identical results.

Notes for reviewers: u3r_met returns c3_w, so +check on atoms of ≥ 2³² blocks is still constrained upstream of this file; and the SoftBLAS gemm reads B with ldc instead of ldb (harmless here since the jet passes ldb == ldc, but worth fixing in SoftBLAS). SoftBLAS's f_gt/f_ge/f_min/f_max macros are NaN-incorrect for the same !le reason and deserve an upstream fix there too; this PR routes around them locally.

🤖 Generated with Claude Code

@sigilante
sigilante requested a review from a team as a code owner July 7, 2026 20:45
@sigilante
sigilante force-pushed the sigilante/lagoon-check-fixes branch from 8356793 to 7ab226f Compare July 7, 2026 21:06
sigilante added a commit to sigilante/urbit that referenced this pull request Jul 7, 2026
Three latent bugs in untested arms, found while auditing the lagoon
jets against this library (urbit/vere#1057):

* +diag built its result with (turn (flop (gulf 0 (dec n))) ...) and
  so returned the diagonal REVERSED.
* +el-wise-op (whose only user is +abs) mapped over (flop (ravel a)),
  reversing the element order.  The comment says the flop is to
  "compensate for LSB", but there is nothing to compensate: +rep
  places the first list element at the low end and +ravel yields
  row-major order, so the round trip already preserves element order
  (as +bin-op, which has no flop, demonstrates).
* +range's bloq-4 (%rh) branch is missing the
  =.  shape.meta  ~[(lent bad)]
  that its 5/6/7 siblings have, so it returned a ray whose shape is
  whatever the caller passed in .meta rather than the element count.

Note on jet skew: the +abs jet in vere has always returned natural
(unreversed) order, so jetted and unjetted ships already disagree
about +abs today -- this change converges the Hoon with the deployed
jet.  The +diag jet crashed unconditionally whenever it fired, and
the +range jet recomputed the count its own way, so there is no
deployed behavior to preserve for those.  urbit/vere#1057 fixes the
jets to match this change exactly; the two should land together.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompted by ~dozreg-toplud's refcount linter, which flagged leaks in
_check, plus a hand audit of the surrounding code.  The fixes fall
into four groups.

_check helper:
* take [meta data] as two borrowed nouns instead of consing a cell at
  every call site (the cell leaked, and it silently assumed ownership
  of the borrowed sample legs); drop the leaked u3qa_dec result
* stop punning raw c3_d/c3_w values as nouns: the shape product was
  truncated through a u3_atom and compared against a noun, so any ray
  with 2^31 or more elements failed _check spuriously (and a crafted
  overflowing shape could pass it)
* compute the product with 64-bit overflow detection; an overflowed
  product cannot match the block count of a real atom, mirroring the
  Hoon exactly
* validate the bloq before calling u3r_met, which is UB at or above
  block size 37 (release builds compile the guard out); at such sizes
  any nonzero atom is one block
* bail %exit only where +check itself crashes (cell dims, improper
  shape list, zero data underflowing +dec)

wrapper/Hoon divergences (jet computed where the Nock crashes, or
vice versa, or produced a different noun):
* transpose/diag passed the whole core to _check, reading the gate's
  battery as ray metadata -- those jets unconditionally bailed %exit
  whenever they fired
* the elementwise (+add/+sub/+mul/+div/+mod), comparison
  (+gth/+gte/+lth/+lte), scalar (+add-scalar &c), +abs, and +trace
  wrappers never called _check at all, though the Hoon asserts it
* +ravel has no +check and never reads the kind, so its wrapper now
  punts (not bails) on an inconsistent ray
* +dot only asserts equal shapes outside the %i754 path; the full
  meta equality and consistency checks moved inside the %i754 case
* +mmul never compared the two rays' bloq/kind, and its gemm switch
  silently returned a zero matrix for an out-of-range bloq; it now
  punts on both
* reduction results (+cumsum/+min/+max/+dot) hard-coded shapes ~[1]
  or ~[1 1] or ~[n 1]; +scalar-to-ray gives an all-ones shape of the
  input's rank
* _set_rounding returns c3n on an unrecognized mode and callers punt,
  instead of bailing %fail

kernel bugs (wrong results, previously unobservable because the
wrapper bailed or untested):
* trace computed (dot d d) -- the sum of SQUARES of the diagonal --
  instead of (cumsum (diag a)); identity-matrix tests masked it
* transpose read and wrote with rows/columns swapped, correct only
  for square inputs, and the wrapper did not swap the result shape
* argmin/argmax returned (len - i - 1) instead of the ravel index i
* diag and abs return elements in natural order, in lockstep with
  urbit/urbit#7388, which removes the erroneous flops in +diag and
  +el-wise-op (the old +abs jet already returned natural order, so
  jetted and unjetted ships disagreed about +abs; the diag jet never
  returned at all)
* trace honors the door rounding mode; linspace fences an indirect
  count atom before using it as a raw integer
* mod (ray and scalar) rounded the quotient with a hardcoded
  round-toward-zero; +toi rounds in the door mode (7 mod 2 is -1
  under %n, not 1).  It also returned values where the Hoon crashes:
  a non-finite quotient (zero or NaN divisor) is (need ~).  And
  mod-scalar/div-scalar multiplied by a rounded 1/n instead of
  dividing -- wrong even for exact quotients (21 mod 7 gave 7,
  21/7 gave 2.9999998).  All four now divide directly, round in
  softfloat_roundingMode, and bail %exit on a non-finite quotient
* gth/gte used SoftBLAS's f_gt/f_ge macros, which are (!le)/(!lt)
  and thus TRUE whenever either operand is NaN; IEEE gt/ge are
  false.  Same disease via f_min/f_max in min/max/argmax.  All now
  use lt/le with swapped operands, which also reproduces the Hoon
  reel-fold exactly (NaN in the head is sticky, interior NaN is
  skipped, first-of-ties wins)
* range counted elements with one-shot ceil((b-a)/d), but the Hoon
  iterates x+d in the door mode, testing each sum against b -- for
  d=.0.1 the shapes disagreed outright (10 vs 11).  The kernel now
  replicates the iteration (values are the accumulated sums) and
  the wrapper takes the count from the result's block count; it
  punts on non-finite bounds/step, a zero step, or a stalled
  accumulator, where the Nock loops forever

leaks:
* the _check cell at every call site; u3qa_dec inside _check;
  _get_dims arrays in the diag/dot wrappers and trace kernel (plus
  raw c3_d dimensions used directly as nouns there); argmin/argmax
  scan buffers; linspace/range result-shape overkeeps

Verified on a fresh fake ship against the base lagoon (with the
urbit/urbit#7388 fixes applied), differentially against a de-jetted
copy of the same library: 36/36 probes agree, including transpose of
non-square, trace, diag, reduction shapes at rank 1, argmin/argmax
with NaN at and after the head, mod under %n/%u, mod-scalar and
div-scalar on exact quotients, gth/gte/lth against NaN, range with
d=.0.1 and a negative step, and add/cumsum/mod-scalar/range under
non-default rounding modes; zero-divisor mod crashes on both doors.
A 40x40 %i754 mmul runs ~3s jetted vs ~25s interpreted, confirming
the jets fire.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sigilante
sigilante force-pushed the sigilante/lagoon-check-fixes branch from 7ab226f to ff06e8a Compare July 7, 2026 23:32
sigilante added a commit to sigilante/urbit that referenced this pull request Jul 8, 2026
Twenty-two oracles for behaviors that have broken before, in this
library or in its vere jets: non-square transpose, diag order and
trace-sums-not-squares, abs element order, reduction result rank,
argmin/argmax indices and NaN stickiness, NaN comparisons, non-square
mmul, %mod door-mode rounding and zero-divisor crash, exact scalar
mod/div quotients, range accumulation count/values (ship-verified
under the %z bare-la default) and its bloq-4 result shape, and
linspace n=1.

They pass identically interpreted and jetted; a jet that disagrees
with any of them is wrong.  Verified green on a fake ship running the
urbit/vere#1057 jets over this branch's library.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sigilante added a commit to urbit/numerics that referenced this pull request Jul 8, 2026
Ports the full urbit/vere#1057 fix set (validation, refcounts, NaN
comparisons, kernel results -- see that PR) into the %int2-extended
32-bit copy and the stateless-SoftBLAS 64-bit copy, preserving the
int2/twoc code and the _la_rnd plumbing.  One deliberate divergence
from the reference: %mod here follows THIS desk's C-fmod semantics
(quotient truncated %z regardless of door mode; canonical quiet NaN,
not a crash, on a zero divisor or non-finite quotient), per the desk's
own "Vere follow-on" note.  The int2 dot case also gains the
meta-sing and consistency checks +bin-op asserts.

tests/lib/lagoon-jet-parity encodes 22 regression oracles for
everything that broke (transpose/diag/trace/abs order and values,
reduction ranks, argmin/argmax NaN behavior, NaN comparisons, exact
scalar mod/div, desk-fmod %mod, range accumulation and bloq-4 shape,
linspace n=1) -- green interpreted against this desk; running it on a
jetted numerics binary is the port's acceptance test.  A stock-
semantics twin lives upstream in urbit/urbit#7388 tests/lib/lagoon.

Also corrects the +la doccords: bare la rounds %z (the bunt of
$rounding-mode), not %n.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sigilante added a commit to urbit/numerics that referenced this pull request Jul 12, 2026
lagoon: jet parity with urbit/vere#1057 — Hoon fixes, C ports, regression suite
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant