fix(ir): the analyst does not count a word as a change (ENG-743)

[henokteixeira]

Linear: ENG-743

Summary

R11 from the open round of 03/09. No new issue.

Session P02 (006189f9…), stretch 5 (Ruth 1:9a — Map Proposition 5: rest (menucha) … each in
the house of her husband). The team said "…sejam felizes, cada uma na casa do seu novo
marido". The analyst marked "felizes" and "novo" as addition — neither an event, actor,
cause, pairing, or outside detail, which is the very definition of addition. Five
re-recordings of a stretch that already told the story right.

The correction verifier's prompt already has this calibration ("Wording varies. Content does
not. This is the whole of your calibration."): different words for the same referent, order,
amount of detail, paraphrase — none of that is a difference. The analyst's prompt never had
this section; it compares the telling-back directly against the map, not one retelling against
another, so the text was adapted (its own example, anchored on the map, not on an "earlier
telling").

What changed

Only the analyst's prompt — the new calibration section and one sentence in the definition of
addition — and one prose test that pins both anchors (house convention: see
test_the_analyst_is_told_where_a_missing_element_sits, its neighbor in the same file). The
parser, the verifier and the closing blocks were not touched (sibling slice
henok/a-missing-part-names-both-voices touches closing_block/CLOSING_*; git merge-tree
clean against it).

Prompt diff, for product-owner review

--- a/app/services/internalization_room/prompts/backtranslation_analysis_system_prompt.md
+++ b/app/services/internalization_room/prompts/backtranslation_analysis_system_prompt.md
@@ -12,6 +12,29 @@ You never talk to the team. You return JSON; someone warmer speaks for you.
 **You know only the telling-back, never the recording itself.** Every finding is about what was
 (or wasn't) in the telling-back. You must never claim to know what the recording says.
 
+## Wording varies. Content does not. This is the whole of your calibration.
+
+The team tells back by **speaking**, and what you read was transcribed by an imperfect ear. No
+honest telling-back repeats the map's own words. Expect all of this, and never report any of it
+as `"addition"` or `"missing"`:
+
+- a different word for the same thing — a synonym, an adjective, an epithet, a name in place of
+  a description, for the same person, place, or thing;
+- a different order of sentences, a different starting point, a different amount of detail;
+- fragments, false starts, repetition, agrammatical {{SESSION_LANGUAGE}}, near-spellings and
+  plausible mishearings of the map's names;
+- a paraphrase that states the same proposition the map gives, in other words.
+
+Here is what that looks like. The map: *rest (menucha) … each in the house of her husband.* The
+telling-back: *"…may they be happy, each in the house of her new husband."* "New" is an adjective
+on "husband"; "happy" is the same blessing said in other words. Neither adds an event, an actor,
+a cause, a pairing, or a detail the map does not give — this proposition gets no finding at all,
+and certainly not `"addition"`.
+
+Judging by literal difference would turn every honest telling-back into a stream of false
+findings, and send the team back to re-record a stretch that already told the story right. When
+wording changed and content did not, report nothing.
+
 ## What to check
 
 Walk the map's material for {{SCOPE}} — every person, place, object, time, event, and marked
@@ -22,7 +45,9 @@ detail — against the telling-back:
    or assume between chunks. Be charitable with names: near-spellings and plausible mishearings
    of the map's names count as present (the telling was transcribed by an imperfect ear).
 2. **Added** (`"addition"`): something the telling-back states that the map does not tell —
-   a name, a cause, a pairing, any outside detail. Quote it briefly in the note.
+   a name, a cause, a pairing, any outside detail. Quote it briefly in the note. A synonym, an
+   adjective, or a paraphrase of a proposition the map already gives is not something outside
+   the map — see the calibration above.
 3. **Meaning changed** (`"meaning_change"`): the telling-back states something the map tells
    differently — not merely absent or extra, but altered in what it means.

⚠️ ir_prompts overrides the file in seeded environments

get_prompt_text reads the ir_prompts row for bt_analyst before the .md file. There is no
seeded row for bt_analyst in tripod_db today (select key from ir_prompts where key='bt_analyst' → 0 rows), so this merge changes behavior immediately, with no row update
needed — as #316 already recorded for the same prompt.

Probe on a real pair (Testing Plan, cases 1 and 2)

analyse_telling_back run for real inside tripod_backend (real credits), over the 10 active
stretches of session P02 006189f9…, with stretch 5 at its 12:55:40 version (4b49b16f…,
"sejam felizes... novo marido") and stretch 1 at the version active at that moment (5f3093b4…,
before the 12:59 re-recording) — the rest at the only version they ever had. scope/
pericope_num = P02, as the session itself stores it (back_translation.scope).

Case 1 — BEFORE (base prompt), 3 of 3 runs reproduce addition

{"evidence_sufficient": true, "findings": [{"kind": "addition", "note": "O relato acrescenta que Noemi desejou que as noras fossem 'felizes'.", "segment_id": "4b49b16f-e0d0-4049-bfd1-3dc6b1026059"}]}

{"evidence_sufficient": true, "findings": [{"kind": "addition", "note": "O relato acrescenta que Noemi desejou que as noras fossem 'felizes'.", "segment_id": "4b49b16f-e0d0-4049-bfd1-3dc6b1026059"}]}

{"evidence_sufficient": true, "findings": [{"kind": "missing", "note": "O relato não menciona que Noemi se levantou (ou se preparou) para sair de Moabe.", "segment_id": "5f3093b4-b9cc-4e5b-84e4-51f8209e2615"}, {"kind": "addition", "note": "O relato acrescenta que Noemi deseja que as noras 'sejam felizes'.", "segment_id": "4b49b16f-e0d0-4049-bfd1-3dc6b1026059"}]}

Case 1 — AFTER (this PR's prompt), 0 of 3 runs with any finding

{"evidence_sufficient": true, "findings": []}

{"evidence_sufficient": true, "findings": []}

{"evidence_sufficient": true, "findings": []}

Plan's criterion: before, reproduces addition for "felizes"/"novo" in ≥1 of 3 (came out 3 of
3); after, 0 of 3 addition and 0 missing for Proposition 5 on stretch 5 (came out 0 of 3, no
finding at all). Closed without tuning anything beyond the text already shown above.

Case 2 — the calibration does not blind the analyst (this PR's prompt, same 10 stretches, only stretch 5 changes)

Without the blessing (stretch 5 = "Noemi beijou as noras e elas choraram."), 3 of 3
missing for Proposition 5:

{"evidence_sufficient": true, "findings": [{"kind": "missing", "note": "falta o conteúdo da segunda bênção de Noemi, pedindo que o Senhor lhes conceda descanso na casa de um marido", "segment_id": "4b49b16f-e0d0-4049-bfd1-3dc6b1026059"}]}

(identical across the 3 runs)

With an outside actor (stretch 5 = "...cada uma na casa do seu marido, e Boaz estava lá."),
3 of 3 addition:

{"evidence_sufficient": true, "findings": [{"kind": "addition", "note": "O relato afirma que 'Boaz estava lá', um detalhe que não consta no mapa para este trecho.", "segment_id": "4b49b16f-e0d0-4049-bfd1-3dc6b1026059"}, {"kind": "missing", "note": "Falta o beijo de despedida de Noemi e o choro inicial das noras após a primeira bênção.", "segment_id": "ff9d72e9-5e51-4a3f-a6eb-88fd9200f167"}]}

(identical across the 3 runs; the extra missing is expected — the override text is shorter
than the real stretch 5 and cuts the kiss/weeping the real version also carries; not what case 2
measures)

Test plan

• New prose test test_the_analyst_does_not_count_a_word_as_a_change
  (tests/test_internalization_room_back_translation.py), house convention: pins the anchor
  "Wording varies. Content does not." across the whole prompt, and isolates item 2
  (Added/addition) from the numbered list by regex — not by split("\n\n"), because the
  whole list (items 1-8) has no blank line between items, and the calibration section itself
  cites "adjective" in its example. Confirmed by simulation (self-review): removing only the
  sentence in item 2 while keeping the calibration makes the test fail; git diff restored by
  copy afterward.
• RED before the change: 1 failed (the anchor did not exist). Confirmed.
• File tests/test_internalization_room_back_translation.py — 68 passed.
• Full suite: 2485 passed, 5 skipped, 1 xfailed (env -u DATABASE_URL, with
  JWT_SECRET_KEY and a sqlite DATABASE_URL on the command).
• ruff check . — All checks passed; ruff format --check . — clean.
• mypy app — Success: no issues found in 597 source files.
• Real probes pasted above (Testing Plan cases 1 and 2).

For the reviewer

• Self-review (subagent) found a non-blocking edge case: the new calibration rule ("a name in
  place of a description... for the same person, place, or thing") could, in theory, collide
  with preservation_violation (category 6) — a map that deliberately withholds a name (as
  P08-Ruth-3-1-5.md does with "our kinsman (moda)" in place of "redeemer") depends on category
  6 continuing to hold over the new calibration. Category 6's text was not touched and should
  prevail in practice, but it is a real interaction in the corpus, not a hypothetical one —
  worth a look from the product owner.
• Adjacent, pre-existing finding, out of scope for this PR: seed_room_prompts is never called
  on this branch (there is no ir_prompts row for any Room prompt today), which is why editing
  the .md is already effective — origin/main already resolved that dead indirection
  (7b1323f1/f4b7ed75, ancestors of main but not of this branch). Not touched here; whoever
  reconciles this line of work with main will run into it regardless.

PR against henok/every-reading-leaves-a-trace (top of the stack, #318). No merge.

[linear-code]

ENG-743

[little-joao]

"When wording changed and content did not, report nothing."

The bullets above scope this to "addition" or "missing", but this closing sentence does not, and the section title says it is "the whole of your calibration". Category 6 is exactly a case where the wording changed and the content did not: a map that withholds "redeemer" behind "our kinsman (moda)" and a telling-back that says the name is literally "a name in place of a description, for the same person" from the first bullet. Maybe scoping this last sentence the same way the bullet intro already does is enough — could you verify that against P08-Ruth-3-1-5.md? You spotted this one yourself in the description, I think it deserves the one-line fix and not only a note for the owner.

[little-joao]

"- a different order of sentences, a different starting point, a different amount of detail;"

"a different amount of detail" under a "never report as missing" list feels too wide for me. Item 1 defines missing as an element the map gives that appears in NO chunk — a telling-back that gives less detail is precisely that, so the analyst ends up with two rules pointing opposite ways. Your case 2 probe cut the whole proposition, which is absence, not a thinner telling. Just make sure here that this bullet is about the map's phrasing being richer than the team's, not about the telling-back carrying fewer elements.

[little-joao]

This why is good but CLAUDE.md asks for it in the docstring instead of a # comment, and the test already has a docstring where it would be read.

[little-joao]

nothing blocking, some points inline