lessons.md

Lessons

Rules for future sessions in this repo, captured after corrections and mistakes.

Working files

• All temp/working files go inside the morpheus folder (tmp/, git-ignored) — never in ~/.lmstudio/scratchpads. (User correction, 2026-10-07.)

Shell / tooling

• The stemlib noun sources are named nom01…nom36 (no dot): the glob stemsrc/*.nom* silently matches only lsj.nom, nom.biblical, … Use stemsrc/nom* when searching all of them.
• zsh chokes on unquoted parentheses in command lists; quote patterns or use a script file for anything with ( ).
• zsh equals-expansion breaks unquoted =… tokens: echo ===MARKER=== dies with "not found" (zsh tries to expand =word as a command path). Quote every literal marker in shell one-liners.
• When scrubbing a name from the repo, grep case-insensitively with word boundaries over ALL file types — not just .py/.md/.sh. Build artifacts (e.g. stemlib/Greek/conjfile, generated by concatenating stemsrc), generated reports, and saved snapshots all retain old strings long after the sources are clean. Fix at the root: edit the source, regenerate the artifact, delete stale snapshots.
• Isolate a code change from snapshot drift with a stash A/B run (Round 8): when a saved evaluation snapshot no longer matches after your change, re-run it with the change git stashed — if the stashed run drifts identically, the difference is environmental/index noise (see endtable lesson), not yours; diff the two runs against each other to prove neutrality.

Beta code / Morpheus internals

• cruncher only matches lowercase beta code; uppercase input (reference- corpus style) silently returns nothing. The toolkit now normalizes this (runner._normalize_beta); keep it that way and test both cases.

• to_beta() drops a breathing when the Unicode letter has none: plain Ι gives *israhl, but Ἰ (U+1F08) gives *)israh/l. When verifying name entries, test with properly breathed/accented forms AND check -n (accent-insensitive) mode.

• Explicit :wd: noun/name forms match exactly (diacritics included); the accent-insensitive pass (cruncher -S -n) is what makes unaccented variants hit.

• Generative :no:<stem> os_ou … patterns do not work for every stem shape (worked for σάββατον, produced nothing for τρυβλίον/ἀδελφιδός). Verify each new entry against the built index (steminds/nomind) and fall back to explicit :wd: lines.

• The incremental stemlib makefile does not always pick up a brand-new stemsrc file; if entries are missing from nomind/vbind, check timestamps and force the target (e.g. touch stemsrc/lsj.nom && make).

• Paradigm declarations (;xx, @ xx) in vbs sources never reach the built index. The build is cat … > conjfile; do_conj; mv conjfile.short /tmp/vbmorph; indexvbs — indexvbs indexes the do_conj output, and only processes :vs/:aj/:no/:vb/:wd/:de: lines (all ;, @, --prefixed lines are ignored). So ;pp/;pf declarations change nothing in vbind; use explicit :vb: lines.

• The generative deriv path rejects non-reduplicated perfect stems (derivio.c checkcomderiv2: if (!had_redupl && Is_perfect(stemtype)) continue;), so -έω pp participles (εὐλογημένος) never generate. Explicit :vb: lines have no such gate.

• GenIrregForm requires a real stemtype in the keys: o_stem alone is derivtypes-only and fails with "no stemtype seen"; add e.g. omi_pr before it (precedent: existing :vb:di/dw omi_pr …). FixRecAcc/FixPersAcc preserve any accent already present, so a :vb:/:wd: line storing the corpus's exact surface spelling round-trips exactly through CheckGenWords.

• standword() rewrites GRAVE→ACUTE (every position) and strips accents after the first on every input before lookup; CheckGenWords compares accents exactly in normal mode (morphstrcmp, only |≡i equivalence) but stripaccs both sides under IGNORE_ACCENTS. Consequence: a stored word-final grave can never match an accented input — store the acute spelling instead (or both spellings, as with αλληλουία). The reference corpus is internally inconsistent (e.g. νοσσιὰν and νοσσιάν for one form; λημφθήτω unaccented in beta but accented in its Greek column) — index every attested standardized spelling.

• Noun lookup (chckindecl) always strips all accents from the key; nomind keys are accent-stripped and merge all full-form variants sharing a stripped key onto one line (morphstrcmp: |≡i, plus "s1 ends ⇒ equal" quirk). The lookup is therefore accent-blind; only CheckGenWords' final compare is accent-sensitive. Multi-entry lines are fine — the parser iterates all entries.

• C# port has placeholder stubs (chckindecl.cs returns 0; CheckGenWordsFunc uses plain string.Compare, no comptab/accent logic) — don't assume C/C# parity when reasoning about behavior from the .cs files.

• nomind key/entry mechanics (Round 7, cost hours): index lines are <key> <form>:<lemma>:<class>[:features]…; lookup keys carry a class prefix (2=indecl/nom, 3=deriv); the pre-index is <file>.lindex (fixed 12-byte records: 8-byte key + 4-byte LE offset). A :wd:<form> line whose form has NO accent marker gets its form field DROPPED by indexnoms (entry becomes <key> :<lemma>:<class>); forms with / etc. are preserved.

• The real gate is CheckGenWords: generated forms must match the input word; for non-enclitics the compare is accent-exact (accword), and FixPersAcc ADDS an accent to one-syllable stems — so unaccented inputs (elided forms) only match under IGNORE_ACCENTS (-n). That's why nom.elided entries resolve only with -n, leaving default-mode behavior untouched.

• C build on this Mac: link against src/{gener,gkends,gkdict,morphlib, greeklib,anal}/*.a; needs the installed Xcode app's SDK exported as SDKROOT

  • -isysroot $SDKROOT (the CommandLineTools SDK is broken — same workaround as scripts/build.sh). C binaries resolve data via env MORPHLIB=/stemlib and append /Greek/… themselves; run indexnoms with that env set.

• :vb: stemtype tokens must come from rule_files/stemtypes.table, not derivtypes.table (Round 8, cost an hour): the analyzer's key parser (ScanAsciiKeys in morphkeys.c) loads STEMTYPES = stemtypes.table; verb types that exist only in derivtypes.table (anw, airw, …) are silently indexed but fail at analysis time with "GenIrregForm Error: no stemtype seen". The stored full form doubles as the generation stem, so ANY accepted token works — the generator's identity slot reproduces it and CheckGenWords matches (Round-8 vbs.papyri uses emi_aor / ow_pr+o_stem / ew_pr for this reason). Reproduce with: bin/cruncher -S -n <file> on a one-word file — the error prints to stderr.

• Endtable/deriv regeneration is not byte-stable across builds (Round 8): after any stemlib index rebuild, saved AF/LXX snapshots can drift ~±0.5pp POS while top-1 stays identical — even when rebuilding from pristine sources. Before chasing a snapshot "regression", re-run the evaluator on a clean-source rebuild to separate build noise from your own changes; then refresh the snapshot and note it in the round record. (Round 7's byte-identical result held because no index rebuild happened that round.)

• cruncher <name> reads <name>.words, not <name> — name probe files accordingly or it reports "cannot find".

• Latin lemmas carry #N homonym markers that survived normalization (Round 8): the Latin stemlib disambiguates irregulars as sum#1, edo#1, quis#2; the parser's trailing-digit strip left the bare # in Analysis.lemma, so frequency lookups for the most common words in Latin (est→sum) silently scored zero. _TRAILING_HOMONYM is now [\d#]*$ (Greek lemmas contain neither — verified zero occurrences, A/B AF re-run byte-identical). Related: Morpheus's Latin analyzer cannot separate adjectives from nouns (both surface as N unless a special stem type fires), so never build joint lemma×POS tables for Latin from external gold that tags them separately — the Ranker's known-not-seen floor would demote correct readings.

Corpus tooling (annotation pipelines)

• PapyGreek ALDT specifics (Round 8): postag is 9 chars — pos person number tense mood voice GENDER CASE degree; gender precedes case (opposite of MorphGNT's order) and letters are lowercase. The editorial-marker zoo beyond LappEpiDoc brackets: ∼∽∤⋰⋱⸢⸣❨❩<>[]{}() + U+0324, |num:N|/|g:…| pipe spans, plus 〚〛 ⊢⊣ ⸌⸍ ⎴⎵ ⦑⦒ ⸜⸝ ⧼ ? … -. Missing any one silently corrupts forms (fake misses); keep ʼ ’ (real elision) and combining diacritics.
• Morpheus.analyze_beta() returns TokenResult wrappers, not raw analysis lists (result.analyses). The runner-level method returns bare lists; the API level does not.
• Reference-corpus parse codes are position-dependent by lexical class: verb codes are 5 chars (tense voice mood person number), nominal codes only 3 (case number gender). Any field-by-field comparison must branch on class AND use per-class minimum lengths, or most nominals silently drop out of the comparison.
• Morpheus's 8-char parsing code contains dashes (3AAI-S--), so shape-based column detection ("parse = the dash-free [A-Z0-9]+ field") misparses generated files — it reads the parse code as the lemma. Parse generated output positionally.
• When training disambiguation models on a gold corpus and then evaluating agreement with that same corpus, say so in the report: the numbers measure domain consistency, not absolute accuracy.
• Never hardcode the reference-corpus location (path or name) in any file. Scripts take --corpus-dir / env (LXX_CORPUS_DIR, AF_CORPUS_DIR); generated reports and bundled tables use a neutral label ("Apostolic Fathers reference corpus", LXX precedent). The user greps the working tree — committed files, tmp/ scratch, and build artifacts must all stay clean.
• A lemma-keyed convention table needs an M==0 guard, not just single-target + frequency (Round 7): form-dependent splits — gender pairs (κύριος/κυρία), enclitic compounds (ἐγώ/κἀγώ) — pass "all mismatching tokens map to one target" but rewriting every occurrence of the source lemma breaks tokens where Morpheus and the gold already agree. Require that no trustworthy token with that choice agrees with the gold, and print rejected pairs (with reasons) in the summary so the table stays auditable.
• Viterbi vs top-1 is a real trade on AF (Round 7): the tagger costs ~1.4pp of lemma (MorphGNT subset) for +1.0pp POS — unlike LXX, where it was net positive. For corpora whose primary column is lemma, default to top-1 and expose --tagger (matching agreement_report.py's flag semantics).
• After toolkit changes, re-run saved evaluators and diff against snapshots (Round 7): the matches_lemma hardening normalized gold that callers already pre-normalized — the ev2 snapshots came back byte-identical, proving no behavioral drift. Keep snapshot files in tmp/ for exactly this purpose.
• When porting a pipeline across repos, diff outputs as the test (Round 7): the AF repo's analyze.py mirrors morpheus's build_morph_af.py; all 15 generated files verified byte-identical before shipping. In the consuming repo, derive tables at run time from its own gold (no stale bundled copies) and pin the minimum upstream commit in the README.
• The reference corpus is UPPERCASE beta code with its own dialect quirks (Round 6): breathings follow the letter (A)PE..., not APE...), / is a plain syllable separator inside words (KURI/OU = κυρίου — substring greps like RIOU miss it), and \, =, | are accent markers used inconsistently for the same word (αὐτοῦ appears as AU)TOU=, AU)TOU\S, AU)TOU/S). Never assume a spelling: key everything through canon() (to_beta∘from_beta + lowercase), and remember one word can occupy several canon keys that must all be covered by any table.
• The eval's "top lemma disagreements" list is misleading (Round 6): it sorts by total form frequency and prints only the first disagreeing line, so a rare quirk on a frequent form looks like a mass error (ὅτι read as ×4,044; actually 7/4044). Always verify with true per-form mismatch counts before sizing a "prize pool".
• Joint-frequency fallback: score unattested (lemma, POS) pairs at a floor of one occurrence — never zero, never full plain frequency (Round 6): the reference's type scheme and Morpheus's POS disagree for the same lemma (αὐτός A vs RD), so zero kills correct candidates; full plain frequency over-credits them (σός(N) 0.432 vs σύ(RP) 0.838 left a gap bigram context flips). An unattested pair can never be the gold reading of any token, so the floor only removes wrong answers.
• The Viterbi tagger keeps one candidate per coarse POS (first on score ties) (Round 6): feature-level ambiguity within a POS — masc/neut homographs, duals — is resolved arbitrarily, and candidates with empty parse codes pass field comparison vacuously (no comparable pairs ⇒ True). Break same-POS ties explicitly (per-spelling reference aggregates; the corpus's accent notation encodes readings) instead of trusting Morpheus's internal candidate order.
• Analysis.lemma/.variants() are Unicode, frequency-table keys normalized Unicode (Round 6): compare via normalize_lemma(), never raw literal equality — from_beta can return NFD and precomposed Greek literals won't match it.
• PROIEL CoNLL-X quirks (Round 8): Greek gold lemmas carry #1 homonym suffixes and optional-letter parens (οὕτω(ς)) — normalize both before comparison or you overstate mismatches (~2.5pp artifact). Latin Morpheus lemmas carry bare # markers — strip #\d*$. CoNLL-X exports also omit unannotated sections (greek-nt: HEB 13 + Catholic epistles; 132,366 of 140,773 XML tokens).
• PROIEL lemma conventions differ from SBLGNT/Morpheus (Round 8): pronoun families headed by the plural (ὑμεῖς/ἡμεῖς vs σύ/ἐγώ), εἶπον kept under λέγω, lowercase σατανᾶς, iota Ἡλίας where SBLGNT has eta Ἠλίας. Cross-corpus "errors" must be split convention-vs-true before sizing fixes (~3,100 tokens ≈ 2.4pp of the PROIEL gap were pure convention).
• License footers lie (Round 8): the Digital Classicist wiki footer said CC BY for the Index Thomisticus Treebank; the project site states CC BY-NC-SA 3.0. Always check the authoritative project page, not aggregator wikis — it decided between two candidate Latin table sources.
• AGLDT treebank format (Round 8): words are <word form=… lemma=… postag=9chars/> elements; lemmas carry Perseus-style trailing homonym digits (for1) which normalize_lemma() already strips. Latin tagset position 1: r=adposition, p=pronoun (opposite of PROIEL UPOS R/P); Greek v2.1 uses Unicode lemmas with initial capitals (normalized lower-case). One reader (--format agldt) serves both languages; filter non-letter keys (-que, heu!, bracketed restorations) and skip postag u.

Python

• Augmented assignment on a function call is a SyntaxError: d.setdefault(k, set()) |= x — use d[k] |= x with a defaultdict instead.
• When writing generators that emit build inputs (stemlib files), make re-runs idempotent by merging with the existing output; otherwise entries generated in run N are dropped in run N+1 once the tool's own output becomes "recognized" input. Use an explicit managed-section marker so curated content is never rewritten.
• Managed-section trap (hit for real, Round 2): a generator that rewrites its whole output file from its own lists will silently wipe anything hand-added above the marker. Keep all curated entries in the script's data structures — never hand-edit the managed region of the generated file. When normalizing emitted values (e.g. grave→acute), apply the normalization at both entry points (new data and load_existing() merge) or re-runs won't converge.