fix(ir): the correction check counts what the stretch carried before judging (ENG-744)

[henokteixeira]

Linear: ENG-744

Summary

R4 finding from the 09/02 afternoon round of the open flow. No issue.

The verification of a correction asked, all at once, "did answering the finding break
anything?" — and a reader answers that question well in one direction only. It confirms
what the retelling says; it walks straight past the clause that dropped, because an
absence leaves nothing on the page to notice.

Measured in the room (session b0cceac5…, Ruth 1:15-18, 09/02 15:47-15:58): stretch 4
(34c05876) said "Onde quer que você morrer, morrerei eu e aí serei sepultada". The
team retold the whole stretch to answer the oath finding; the new version
(aab97a9d) gained the oath and lost the burial. verify_correction ran
over that pair and returned a meaning_change — and no missing. The loss only surfaced
two turns later, at the full reread, pointed at chunk 3. The same content ended up
said in both 3 and 4, and the user saw "redundancy".

The session's own history shows the sequence (the guide's utterances, in order):

"…o juramento de Rute, onde ela chama o nome de Javé e pede que ele a castigue se ela quebrar a promessa, não apareceu."
"…Noemi continuou a insistir para que Rute voltasse — mas a história conta que … Noemi ficou em silêncio."
"…o voto de Rute sobre morrer onde Noemi morrer e ser sepultada no mesmo lugar não apareceu."

The research that motivated the design (arXiv 2608.31016) says LLM judges detect addition and
change well (0.79–0.94) and omission close to chance (0.50–0.63), and that what reverses it is
restructuring the task: enumerating the source elements first and checking them one by one.

What changed

1. The correction prompt asks for the enumeration before the judgement: the reader lists what
   the stretch's earlier count carried, element by element, and marks each one as
   still stated or not (carried, with element and still_told). The output JSON carries
   carried before resolved and findings.
2. The parser derives the loss. An entry marked as not stated is a missing on this
   stretch, whether or not it exists in the model's findings list. Expecting the model to also
   report it would be the holistic question again — the one that let the burial through.
3. Dedupe. A derived missing is suppressed when some missing the model
   wrote already carries, in its note, every content word of the element (≥4 letters,
   unaccented, lowercase). A deliberately strict rule: the two errors are not equal —
   counting twice costs a re-recording; suppressing wrongly means never asking.
   An element with no content word covers nothing (the empty set is contained in every
   note, and the room would discard every loss it counted).
4. Compatibility. A reply with no carried is read exactly as it always was.

findings_after_correction did not change: the dedupe happens earlier, in the parser, so the list
that reaches it is already the room's decision.

⚠️ This does not change the room by itself

get_prompt_text reads the ir_prompts row and only falls back to the .md file when there is no row —
and seed_room_prompts inserts one per key. Editing the file does not change the prompt in
production. The text below is what goes to the product owner's review; the row is updated
separately. The tests use the file (empty SQLite), which is why the compatibility case
exists: both reply shapes stay live at the same time.

Prompt before/after (for the product owner's review)

diff --git a/app/services/internalization_room/prompts/backtranslation_correction_system_prompt.md b/app/services/internalization_room/prompts/backtranslation_correction_system_prompt.md
index 038b3b7c..8e120284 100644
--- a/app/services/internalization_room/prompts/backtranslation_correction_system_prompt.md
+++ b/app/services/internalization_room/prompts/backtranslation_correction_system_prompt.md
@@ -45,7 +45,26 @@ outcome available. When wording changed and content did not, you report nothing.
 
 ## What to decide
 
-**1. Was the finding answered?** (`resolved`)
+**1. What did the earlier telling carry?** (`carried`)
+
+Before you judge anything, count. List the elements of the Meaning Map that the **earlier**
+telling of this stretch states — one entry each, in {{SESSION_LANGUAGE}}, written as the short
+phrase you would use to name that element to the team. Not the map's wording, not a whole
+sentence, and only elements the map gives for {{SCOPE}}: a detail the team added on their own
+is not one, and neither is an element the map gives that the earlier telling never stated.
+
+Then take each entry on its own and say whether the **new** telling still states it —
+`still_told: true` or `false`. In any words: this is the same content question the whole of
+your calibration above turns on, so other words, another order and another amount of detail
+all still count as stated. Judge each entry against that entry alone, before you have formed
+any view of the retelling as a whole.
+
+Count first because of what counting is for. Read whole, a retelling shows you what it says,
+and you will confirm every one of those and read straight past the one clause that quietly
+went — an absence leaves nothing on the page to notice. Going element by element is what turns
+"is anything missing?" into a question that has an answer.
+
+**2. Was the finding answered?** (`resolved`)
 
 Read the finding, and ask only whether the new telling now carries what it asked for, measured
 against the Meaning Map. `true` when it does; `false` when it does not. A finding about an
@@ -53,7 +72,7 @@ element the map gives is answered when that element is stated in the new telling
 Do not require the team to have said it the way the map says it, and do not require them to have
 mentioned the finding itself.
 
-**2. Did answering it break something?** (`findings`)
+**3. Did answering it break something?** (`findings`)
 
 Compare the new telling against the earlier one and against the map, and report only these:
 
@@ -61,7 +80,10 @@ Compare the new telling against the earlier one and against the map, and report
   and the new one no longer states. This is the regression this whole check exists to catch:
   retelling a stretch can quietly drop something only that stretch carried. Report it only when
   the element is in the map — something the team simply said differently, or a detail the map
-  does not give, is not a loss.
+  does not give, is not a loss. You have already counted these: every entry you marked
+  `still_told: false` is a loss, and the room reads it straight from your count. You do not
+  need to write it again here. If you do, name the element in the note in the same words you
+  gave it in `carried`, so the room can see that the two are one loss and not two.
 - **Added** (`"addition"`): something the new telling states that the map does not tell — a name,
   a cause, a pairing, an outside detail. Quote it briefly in the note.
 - **Meaning changed** (`"meaning_change"`): the new telling states something the map tells
@@ -85,6 +107,9 @@ Return **only** this JSON (no prose, no fences):
 
 ```json
 {
+  "carried": [
+    { "element": "one short phrase, in {{SESSION_LANGUAGE}}, naming an element the earlier telling stated", "still_told": true }
+  ],
   "resolved": true,
   "findings": [
     { "kind": "missing" | "addition" | "meaning_change" | "preservation_violation" | "unclear", "note": "one short sentence, in {{SESSION_LANGUAGE}}, phrased about the telling-back" }
@@ -94,7 +119,8 @@ Return **only** this JSON (no prose, no fences):
 
 `resolved` and `findings` are independent. A correction can answer the finding and still lose an
 element (`true` with findings), or leave the finding standing while breaking nothing (`false`
-with none). A clean correction is `{ "resolved": true, "findings": [] }`.
+with none). A clean correction is a `carried` list whose entries are all `still_told: true`,
+with `{ "resolved": true, "findings": [] }` beside it.
 
 ## The Meaning Map
 

Probe on the real pair (Testing Plan case 7)

verify_correction run for real inside the tripod_backend container over the database
pair (earlier = 34c05876…, corrected = aab97a9d…, the oath finding), with the prompt
before and after. Two rounds, same result. Raw model responses:

BEFORE (base prompt) — no missing

{
  "resolved": true,
  "findings": [
    {
      "kind": "meaning_change",
      "note": "O novo relato diz que Noemi continuou insistindo para Rute voltar, mas o mapa diz que ela parou de falar."
    }
  ]
}

AFTER (this PR's prompt) — the burial comes out as not stated

{
  "carried": [
    {
      "element": "onde você morrer, eu morrerei",
      "still_told": false
    },
    {
      "element": "onde você for sepultada, eu serei sepultada",
      "still_told": false
    }
  ],
  "resolved": true,
  "findings": [
    {
      "kind": "missing",
      "note": "onde você morrer, eu morrerei"
    },
    {
      "kind": "missing",
      "note": "onde você for sepultada, eu serei sepultada"
    },
    {
      "kind": "meaning_change",
      "note": "O relato diz que Noemi continuou a insistir, mas o mapa diz que ela parou de falar."
    }
  ]
}

The model, with the new prompt, enumerated both losses and also reported them in findings —
exactly case 3. Running that real reply through this PR's parser: 3 findings
(2 missing + 1 meaning_change), not 5. The dedupe is what makes the difference.

Test plan

New file tests/test_ir_a_mend_is_checked_against_what_it_carried.py, 7 scenarios, all
through the finish route, black-box. The reader double returns whatever reply the scenario
dictates — the scenarios are about what the room does with it.

JWT_SECRET_KEY=test-secret-for-pytest-only DATABASE_URL="sqlite+aiosqlite:///./test.db" uv run pytest tests/ -q

RED, before the change — 4 failed, 4 passed (measured with the original 8 scenarios)

FAILED test_an_element_counted_as_no_longer_told_is_a_finding_on_that_stretch  (assert 0 == 1)
FAILED test_the_team_is_told_which_element_fell                                (no finding reached the speaker)
FAILED test_a_count_that_cannot_be_read_leaves_the_reported_findings_standing[not-a-list]   (no trace)
FAILED test_a_count_that_cannot_be_read_leaves_the_reported_findings_standing[no-verdict]   (no trace)

The Testing Plan expected case 3 to fail too, before the change. It does not fail, and the plan's
own parenthetical says why: with no derivation there is no duplicate, so the result is already

1. Confirmed with the plan's author; he is the dedupe's guard, and his proof is mutation 2.

Mutations — each scenario has a positive control

mutation	scenarios that fall
1. derivation switched off (if False and "carried" in parsed)	the 4 from RED
2. derivation on, dedupe off	only case 3 — assert 2 == 1
3. still_told sense inverted	element note, clean correction, dedupe, other-kind
4. a reply with no count treated as unreadable	nothing here — 10 scenarios in the neighboring file

Mutations applied over a copy of the file and restored by copy. Mutation 1 was re-measured after
the removal below and still fells the same 4.

One scenario withdrawn: the compatibility case was already covered

The Testing Plan called for a case for "a loss reported without enumeration still holds".
It was written, it passed, and mutation 4 showed where its coverage actually lives: making the
room require the count fells 10 scenarios in
test_ir_a_correction_is_verified_on_its_own — that file's double never emits
a count, so every verification of it is already a reply in the old shape. The case here was
the same guard written twice, and the copy that carries it is the one a reader of the old shape
would look for. Withdrawn at the plan author's request; behavior did not change.

Suites

• the three files together, after the removal — 87 passed
• tests/test_ir_a_correction_is_verified_on_its_own.py + tests/test_internalization_room_back_translation.py — 80 passed
• ruff check — All checks passed!; ruff format --check — 811 files already formatted
• mypy app — Success: no issues found in 597 source files
• Full suite: 2458 passed, 5 skipped, 1 xfailed in 33min09 (measured before the removal; one
  scenario fewer since then) (env -u DATABASE_URL,
  with JWT_SECRET_KEY and DATABASE_URL on the command — without them the plan measured 125
  environment failures, none of them code)

git merge-tree against the neighboring slices

Both clean against this HEAD:

• henok/the-checked-turn-asks-nothing (touches closing_block, which this PR does not touch)
• henok/eng-719-the-parser-keeps-what-it-can-read (touches the same _parse_correction)

For the reviewer

• Semantics, not text: eng-719 replaces the parser's silent return None exits with a
  named condition. This PR's new block returns None with logger.warning. The merge is
  clean, but once both land someone needs to reconcile the two styles — not done here, because
  it is that slice's own scope.
• resolved: false discards what the verification found. findings_after_correction already
  did this before this PR ("what the correction broke does not land on top of a finding that is
  still standing"), and a loss derived on a refused correction is lost the same way.
  Existing, documented behavior; not touched, but worth knowing that the derivation inherits it.
• Order: derived findings come after the ones the model wrote. Since only the first one
  reaches the Speaker, on a turn with a model addition plus a counted loss, the team hears the
  addition first. A least-change choice; if the loss should take priority, it's a one-line fix.

---

Addendum (09/02, evening) — the count had a blind side: what the correction brought back

Finding from today's open round, in a real session (P06, 118e4c47…, ~21:12 UTC): a stretch
corrected to answer a meaning_change — the new version brought back exactly the
missing clause ("junto com as suas servas", P06's map Proposition 1) — came back from
this PR's verify_correction with a false addition, three times in a row, and dropped
the room into fail-safe twice.

Cause: carried only enumerates what the earlier count said. An element the map gives
that the earlier count never had does not appear in carried — it has nowhere to go — and when
comparing the new version against the map at step 3, the reader reads that clause as content that
was "not tracked" and reports it as addition, even with this PR's earlier version already
saying "an element the map does give... is never an addition". The sentence alone changes nothing:
the reader never revisits that rule for a clause carried never listed.

Fix (same spirit as this PR — count before judging, now in both directions):

1. Step 1 of the prompt gained a second list, brought_back: the elements the map gives
   for the scope that the new count states and the earlier one did not — named this way
   because they are the correction arriving, never an addition.
2. The parser accepts the field (a list of strings or of {element}), does not generate a
   finding from it, and — a mechanical belt, in case the reader still slips and reports one of
   them as addition anyway — suppresses that finding, with the same content-word dedupe this
   PR already uses for a loss counted and reported twice (_already_reported).

Prompt diff (on top of what the PR already brought):

@@ -64,6 +64,12 @@
 went — an absence leaves nothing on the page to notice. Going element by element is what turns
 "is anything missing?" into a question that has an answer.
 
+Now list the other direction (`brought_back`): elements the Map gives for {{SCOPE}} that the
+**new** telling states and the **earlier** one did not — one short phrase each, in
+{{SESSION_LANGUAGE}}, the same way you named `carried` entries. These are the correction
+arriving, never an addition: an element listed here must never also appear in `findings` as
+`"addition"`.
+
 **2. Was the finding answered?** (`resolved`)
 
 Read the finding, and ask only whether the new telling now carries what it asked for, measured
@@ -85,7 +91,11 @@
   need to write it again here. If you do, name the element in the note in the same words you
   gave it in `carried`, so the room can see that the two are one loss and not two.
 - **Added** (`"addition"`): something the new telling states that the map does not tell — a name,
-  a cause, a pairing, an outside detail. Quote it briefly in the note.
+  a cause, a pairing, an outside detail. Quote it briefly in the note. An element the map does
+  give for {{SCOPE}} is never an addition, even when the earlier telling did not carry it and
+  `still_told` above is your own report of that: a correction that brings back what the earlier
+  telling was missing is exactly what answering a finding looks like. `carried` exists to find
+  what this stretch **lost**, never to make what it never had into something new.
 - **Meaning changed** (`"meaning_change"`): the new telling states something the map tells
   differently — altered in what it means, not merely absent or extra.
 - **Preservation violated** (`"preservation_violation"`): the new telling makes explicit
@@ -110,6 +120,9 @@
   "carried": [
     { "element": "one short phrase, in {{SESSION_LANGUAGE}}, naming an element the earlier telling stated", "still_told": true }
   ],
+  "brought_back": [
+    "one short phrase, in {{SESSION_LANGUAGE}}, naming an element the new telling states that the earlier one did not"
+  ],
   "resolved": true,
   "findings": [
     { "kind": "missing" | "addition" | "meaning_change" | "preservation_violation" | "unclear", "note": "one short sentence, in {{SESSION_LANGUAGE}}, phrased about the telling-back" }

Real probe, with the corrected prompt — both directions, 3× each

verify_correction run for real (real credits, inside tripod_backend) over two
real pairs:

(a) Today's P06 — earlier = the 21:10:14 stretch ("Boaz fala pra Rute colher espigas em
outros campos e não no dele"), corrected = the 21:12:29 one ("Boaz fala pra Rute colher
espigas somente no campo dele, junto com as suas servas"), finding = message 1's meaning_change.

(b) This PR's Ruth 1 — earlier = 34c05876… ("Onde quer que você morrer, morrerei eu e
aí serei sepultada"), corrected = aab97a9d… ("Que o Senhor me castigue se outra coisa que
não seja a morte me separar de você. Quando Noemi viu que Rute estava mesmo decidida a
acompanhá-la, continuou a insistir pra que ela voltasse"), finding = the oath one (missing:
"o juramento de Rute, onde ela chama o nome de Javé e pede que ele a castigue se ela quebrar a
promessa, não apareceu") — the same session b0cceac5 used in this PR's original probe, the same
finding cited there as "the oath finding".

(a) — 3 of 3 rounds, no addition:

{
  "carried": [{ "element": "Boaz fala para Rute colher espigas", "still_told": true }],
  "brought_back": ["colher somente no campo dele", "junto com as suas servas"],
  "resolved": true,
  "findings": []
}

{
  "carried": [{ "element": "Boaz fala para Rute colher espigas", "still_told": true }],
  "brought_back": ["colher somente no campo dele", "junto com as suas servas"],
  "resolved": true,
  "findings": []
}

{
  "carried": [{ "element": "Boaz fala para Rute colher espigas", "still_told": true }],
  "brought_back": ["colher somente no campo dele", "ficar junto com as servas dele"],
  "resolved": true,
  "findings": []
}

(b) — 3 of 3 rounds, the burial is still reported as lost:

{
  "carried": [
    { "element": "onde você morrer, eu morrerei", "still_told": false },
    { "element": "onde você for sepultada, eu serei", "still_told": false }
  ],
  "brought_back": ["juramento com o nome do Senhor", "pedido de castigo/maldição", "condição de que apenas a morte as separe", "Noemi vê a determinação de Rute"],
  "resolved": true,
  "findings": [
    { "kind": "meaning_change", "note": "O relato diz que Noemi continuou a insistir para Rute voltar, mas o mapa diz que ela parou de falar." }
  ]
}

{
  "carried": [
    { "element": "onde você morrer, eu morrerei", "still_told": false },
    { "element": "onde você for sepultada, eu serei", "still_told": false }
  ],
  "brought_back": ["juramento com o nome do Senhor", "pedido de castigo/maldição", "condição de que apenas a morte as separe", "Noemi vê a determinação de Rute"],
  "resolved": true,
  "findings": [
    { "kind": "meaning_change", "note": "O relato diz que Noemi continuou a insistir para Rute voltar, mas o mapa diz que ela parou de falar." }
  ]
}

{
  "carried": [
    { "element": "onde você morrer, eu morrerei", "still_told": false },
    { "element": "onde você for sepultada, eu serei", "still_told": false }
  ],
  "brought_back": ["juramento de Rute invocando o Senhor", "fórmula de auto-maldição (que o Senhor me castigue)", "condição do juramento (se algo além da morte separar)", "Noemi vê a determinação de Rute"],
  "resolved": true,
  "findings": [
    { "kind": "missing", "note": "onde você morrer, eu morrerei" },
    { "kind": "missing", "note": "onde você for sepultada, eu serei" },
    { "kind": "meaning_change", "note": "O relato diz que Noemi continuou a insistir, mas o mapa diz que ela parou de falar (v. 18)." }
  ]
}

Running all six real responses through this PR's parser (_parse_correction, already with the fix):
(a) 0 of 3 carry addition in the final result; (b) 3 of 3 still carry the burial
as missing (the dedupe between carried and the explicit finding in round 3 still collapses
to a single finding, as already proven by this PR's original probe).

One earlier attempt failed and is recorded here for transparency: the sentence alone in the
Added rule (without the brought_back list) did not close it — 3 of 3 still came back as
addition, because the reader never gets to test that sentence for a clause carried never
listed. That is what motivated the structural fix above instead of just reinforcing the existing
rule.

Verification

• tests/test_ir_a_mend_is_checked_against_what_it_carried.py +
  tests/test_ir_a_correction_is_verified_on_its_own.py +
  tests/test_internalization_room_back_translation.py — 89 passed
• Two new scenarios in the first file: a non-empty brought_back with empty findings is
  a clean correction (did not fail before — the unknown field was already ignored); an addition
  with the same words as an element in brought_back is suppressed (did fail before,
  assert 1 == 0; a mutation removing the filter reproduces the same failure, restored by
  copy).
• ruff check — All checks passed; ruff format --check — clean
• mypy app/services/internalization_room/back_translation.py — Success: no issues found

[linear-code]

ENG-744

[little-joao]

"The same subset-of-content-words test _already_reported uses for a derived loss, mirrored for the opposite mistake"

I am not too sure the mirror is exact here. On _already_reported a false match only stops the room repeating a loss the reader already wrote — the finding still reaches the team. Here a false match deletes the addition and nothing takes its place, which is the cost your own docstring below calls the worse one ("suppressing a real one means they are never asked at all"). And a real addition on the same clause is easily a superset: brought_back = "Orfa é citada pelo nome" against a note like "a equipe diz que Orfa é citada pelo nome do marido, que o mapa não conta" — every content word is there and the real addition goes silent. Could you check if the subset test is tight enough on this side? @henokteixeira

[little-joao]

Lenient is fine, but silent too? Everything else in this parser leaves a trace when a reply comes back in a shape it cannot read, and here the backstop just does not apply — the exact case the addendum describes dropping the room into fail-safe. Maybe a logger.warning like the one in _elements_the_count_lost.

[little-joao]

nothing blocking, two points inline