-
Notifications
You must be signed in to change notification settings - Fork 0
fix(ir): the checked turn asks nothing (ENG-745) #317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
8 commits
Select commit
Hold shift + click to select a range
8b76511
feat(ir): the checked turn's closing carries no question
henokteixeira f695838
Merge commit '8168af12' into tmp-recon-317
henokteixeira 79de81c
fix(alembic): join the two heads main's ir_prompts drop left behind
henokteixeira 5e87a2d
Merge commit '5237cef3' into tmp-fix-317
henokteixeira ca13c41
Merge commit 'b14a55e7' into tmp-del-317
henokteixeira f2169e1
Merge branch 'tmp-309-fechamentos' into tmp-317-fecho
henokteixeira 8881ace
Merge branch 'tmp-309-fechamentos' into tmp-317-fecho
henokteixeira ae80bb5
Merge branch 'r336-309' into r336-317
henokteixeira File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| import pytest | ||
|
|
||
| from app.services.internalization_room.back_translation import ( | ||
| CLOSING_CHECKED, | ||
| CLOSING_PLAIN, | ||
| Finding, | ||
| FindingKind, | ||
| closing_block, | ||
| findings_block, | ||
| segments_block, | ||
| ) | ||
| from app.services.internalization_room.run_turn import run_verdict_turn | ||
| from tests.test_internalization_room_back_translation import ( | ||
| SPEAKER, | ||
| VALIDATOR, | ||
| P, | ||
| _settings, | ||
| _told, | ||
| patch_loop, | ||
| patch_speaker, | ||
| ) | ||
|
|
||
| #: Silences an unused-import lint warning: pytest discovers these fixtures by name because | ||
| #: they are imported into this module's namespace, not because anything here calls them | ||
| #: directly. | ||
| _FIXTURES = (patch_loop, patch_speaker) | ||
|
|
||
| #: The phrase every kind but this one still closes with. Its absence is half of what "asks | ||
| #: nothing" means here. | ||
| ANSWERABLE_QUESTION = "answerable question" | ||
|
|
||
| #: The other half of the old instruction's vocabulary — the word this closing may not use | ||
| #: even if it avoids a literal question mark. | ||
| INVITATION = "invitation" | ||
|
|
||
| #: The prompt's own promise of a next round, previously a static line under `{{CLOSING}}` | ||
| #: on every verdict turn. `CLOSING_CHECKED` has no next round, so this and it may not both | ||
| #: reach the Speaker on the same turn — found in code review, same class of defect as the | ||
| #: original bug: a signal that promises continuation on the one turn that has none. | ||
| CONTINUES_TELLING_BACK = "finish the telling-back again" | ||
|
|
||
|
|
||
| async def _checked_turn_for(draft: str, patch_speaker) -> str: | ||
| """The Speaker's system prompt on a turn with no finding that closes `checked`.""" | ||
| agent = patch_speaker(draft) | ||
| await run_verdict_turn( | ||
| session_language="Portuguese", | ||
| language_code="pt", | ||
| findings_text=findings_block(None), | ||
| closing=closing_block(None, checked=True), | ||
| scope=P, | ||
| pericope_num=P, | ||
| messages=[], | ||
| speaker_prompt=SPEAKER, | ||
| validator_prompt=VALIDATOR, | ||
| settings=_settings(), | ||
| ) | ||
| return str(agent.seen[0]) | ||
|
|
||
|
|
||
| async def _checked_turn_with_loop(draft: str, patch_loop): | ||
| """Same turn, through the draft-and-gate loop so the Validator's own brief is visible.""" | ||
| told = _told() | ||
| agent = patch_loop(draft, told) | ||
| outcome = await run_verdict_turn( | ||
| session_language="Portuguese", | ||
| language_code="pt", | ||
| findings_text=findings_block(None), | ||
| closing=closing_block(None, checked=True), | ||
| scope=P, | ||
| pericope_num=P, | ||
| messages=[], | ||
| telling_back=segments_block(told), | ||
| speaker_prompt=SPEAKER, | ||
| validator_prompt=VALIDATOR, | ||
| settings=_settings(), | ||
| ) | ||
| return outcome, agent | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_a_checked_turn_does_not_ask_a_question(patch_speaker) -> None: | ||
| """Case 1. Sem achado e com evidência suficiente, o fechamento não pede pergunta. | ||
|
|
||
| R5: a rota marcava `checked = True` e ainda assim mandava o narrador terminar com | ||
| "exactly one answerable question or invitation" — o `CLOSING_PLAIN` de sempre. Não há | ||
| próximo turno depois de `checked`, então a pergunta não tinha para quem responder. | ||
| """ | ||
| spoken_to = await _checked_turn_for("A passagem foi contada e conferida.", patch_speaker) | ||
|
|
||
| assert ANSWERABLE_QUESTION not in spoken_to | ||
| assert INVITATION not in spoken_to | ||
| assert "Do not ask" in spoken_to | ||
| assert CLOSING_PLAIN not in spoken_to | ||
| assert CLOSING_CHECKED in spoken_to | ||
| assert CONTINUES_TELLING_BACK not in spoken_to | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_the_validator_is_shown_the_checked_closing_not_the_plain_one( | ||
| patch_loop, | ||
| ) -> None: | ||
| """Case 2. O validador vê o mesmo fechamento que o narrador recebeu, não `CLOSING_PLAIN`.""" | ||
| _, agent = await _checked_turn_with_loop("A passagem foi contada e conferida.", patch_loop) | ||
|
|
||
| assert CLOSING_CHECKED in agent.briefs[0] | ||
| assert CLOSING_PLAIN not in agent.briefs[0] | ||
| assert ANSWERABLE_QUESTION not in agent.briefs[0] | ||
|
|
||
|
|
||
| @pytest.mark.asyncio | ||
| async def test_an_obedient_narrator_passes_the_checked_turn(patch_loop) -> None: | ||
| """Case 3, first half. A draft that only affirms told-and-checked is not refused. | ||
|
|
||
| This only proves the loop still runs end to end on this turn shape — none of the | ||
| Validator double's three rules read the closing at all, so this case cannot tell a | ||
| working `CLOSING_CHECKED` from a broken one either. Cases 1 and 2 carry that weight. | ||
|
|
||
| The mirror half of case 3 — a draft that still asks "Como vocês se sentem?" — is not | ||
| written here. `ValidatorReadsOnlyItsOwnPrompt` judges three things read straight out of | ||
| its own prompt: a claim about the telling-back with no evidence for it, a destination | ||
| the brief never named, and a name the team never said. None of its three rules read the | ||
| closing's own instruction not to ask, so it cannot fail a draft on that basis, and a test | ||
| asserting a fail-safe there would not be testing this change — it would be testing | ||
| against a rule the double does not have. Reported as a gap, not filled with a test that | ||
| would look like coverage of it. | ||
| """ | ||
| outcome, _ = await _checked_turn_with_loop("A passagem foi contada e conferida.", patch_loop) | ||
|
|
||
| assert outcome.used_fail_safe is False | ||
| assert outcome.speech == "A passagem foi contada e conferida." | ||
|
|
||
|
|
||
| def test_insufficient_evidence_keeps_the_plain_closing() -> None: | ||
| """Case 4 (guard). Sem achado mas com evidência insuficiente, o fechamento não muda. | ||
|
|
||
| Há próximo turno — a equipe vai contar mais — então a pergunta continua tendo para quem | ||
| responder. `state.checked` só fica `True` com achado nenhum *e* evidência suficiente; sem | ||
| a segunda metade, o chamador nunca passa `checked=True` para `closing_block`. | ||
| """ | ||
| assert closing_block(None, checked=False) == CLOSING_PLAIN | ||
| assert closing_block(None) == CLOSING_PLAIN | ||
| assert CONTINUES_TELLING_BACK in closing_block(None, checked=False) | ||
|
|
||
|
|
||
| def test_a_finding_ignores_the_checked_flag() -> None: | ||
| """Case 5 (guard). Com achado, nada muda — os fechamentos de achado continuam os de hoje. | ||
|
|
||
| Na rota real `state.checked` só é `True` quando não há achado, mas `checked` é | ||
| ignorado sempre que há um: a passagem não pode estar conferida no mesmo turno em que | ||
| o narrador está falando sobre algo que o analista encontrou. | ||
| """ | ||
| finding = Finding(kind=FindingKind.ADDITION, note="Orfa", segment_id="segmento-2") | ||
|
|
||
| assert closing_block(finding, checked=True) == closing_block(finding, checked=False) | ||
| assert closing_block(finding, checked=True) != CLOSING_CHECKED | ||
| assert CONTINUES_TELLING_BACK in closing_block(finding, checked=True) |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The move itself looks right to me, but the⚠️ operational note on the description does not match what I see here:
On head
get_prompt_textis justreturn default_prompt(key)["prompt"](app/services/internalization_room/prompts.py) — the committed file, no row read at all — andir_promptswas dropped inalembic/versions/20260902_room09_drop_ir_prompts.py. So if I am reading it right the.mdremoval propagates by itself and nobody has to touch a row by hand, and the warning as written sends someone to a table that is not there anymore. Could you verify that and, if it is the case, drop the note from the description?