Skip to content

Commit 8202dce

Browse files
jmjavacursoragent
andcommitted
Add a payload-key rename case to the harder multi-file suite.
Isolate leftover user_id dict-key aliases from acct: prefix drift so the fifth Phase 3 case stays offline-first and layered. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent ffecae3 commit 8202dce

6 files changed

Lines changed: 174 additions & 5 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,8 @@ connected through SSH local forwarding.
110110
```
111111

112112
Harder multi-file cases (exception rename, dataclass field rename, tuple
113-
return + facade, keyword-parameter rename) are offline by default.
113+
return + facade, keyword-parameter rename, payload-key rename) are
114+
offline by default.
114115
`--live` skips with exit 0 when Ollama is down; that skip is not a
115116
quality pass:
116117

‎docs/evaluation-protocol.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ The same scorer runs against:
6464
| `rename_dataclass_field` | `local_refactor` | Rename `Person.years` → `age` in producer and consumer; no property alias |
6565
| `widen_return_keep_facade` | `local_refactor` | `apply_discount` returns `(discounted, saved)`; `line_total` / `savings` stay ints |
6666
| `rename_kwarg_across_files` | `local_refactor` | Rename `send(subject=)` → `send(title=)`; no leftover parameter / `**kwargs` alias |
67+
| `rename_payload_key_across_files` | `local_refactor` | Rename payload key `user_id` → `account_id`; no leftover key / `.get()` alias |
6768
| `implement_clamp` | `local_code` | Implement `clamp` from a spec, no starter file |
6869
| `explain_clamp` | `local_explain` | Prose: names the function and bounds; mentions clipping |
6970
| `review_login` | `local_review` | Prose first-pass: flags None and missing auth. **Not** an apply |
@@ -79,9 +80,11 @@ tell layers apart:
7980
- unified diff only → format pass, structure skipped
8081
- leftover `QuotaError` alias or `years` property → structure fail
8182
- leftover `subject=` parameter / keyword alias → structure fail
83+
- leftover `user_id` payload key / `.get()` fallback → structure fail
8284
- exception threshold or greeting format drift → behavior fail
8385
- `apply_discount` still returns an int → structure fail; wrong `DISCOUNT_PERCENT` → behavior fail
8486
- `send` joins with `|` instead of `:` → behavior fail
87+
- `label` prefixes with `user:` instead of `acct:` → behavior fail
8588

8689
## Measurement harness
8790

@@ -214,7 +217,7 @@ After repeated live runs, a paper may claim:
214217
- Layer-conditional rates on this corpus (format vs structure vs behavior).
215218
- That a vaguer prompt raises structure failures on the same oracle
216219
(`whitespace_extract` vs `whitespace_extract_vague`).
217-
- That leftover type/field/keyword-parameter aliases, catch-site drift,
220+
- That leftover type/field/keyword-parameter/payload-key aliases, catch-site drift,
218221
and a helper signature change that breaks a public facade are
219222
distinguishable layers on the harder multi-file suite.
220223
- That shape-only test generation overstates success relative to

‎docs/phase3-log.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Cloud-safe stub + apply-gate run:
1818
| 2026-09-07 | all local_* eval tools | stub | 10-case corpus + apply gate | accepted / rewritten / rejected per job | Cloud Agent; no GPU; A6 now executes tests |
1919
| 2026-09-10 | local_refactor | fixtures | harder multi-file suite (3 cases) | scorer only | Offline corpus + runner; no live rate; Halo not started |
2020
| 2026-09-10 | local_refactor | fixtures | harder suite + `rename_kwarg_across_files` | scorer only | Fourth case; leftover `subject=` alias vs colon-join drift; no live rate |
21+
| 2026-09-10 | local_refactor | fixtures | harder suite + `rename_payload_key_across_files` | scorer only | Fifth case; leftover `user_id` key / `.get()` alias vs `acct:` prefix drift; no live rate |
2122

2223
Columns:
2324

‎src/local_coding_slm/eval/cases.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,9 @@
3232
RENAME_KWARG_ALIAS,
3333
RENAME_KWARG_PARTIAL,
3434
RENAME_KWARG_SEPARATOR,
35+
RENAME_PAYLOAD_ALIAS,
36+
RENAME_PAYLOAD_PARTIAL,
37+
RENAME_PAYLOAD_PREFIX,
3538
WIDEN_RETURN_INT,
3639
WIDEN_RETURN_PARTIAL,
3740
WIDEN_RETURN_WRONG_RATE,
@@ -579,4 +582,31 @@ class Fixture:
579582
False,
580583
"behavior",
581584
),
585+
Fixture(
586+
"rename_payload_golden",
587+
"rename_payload_key_across_files",
588+
GOLDEN_FOR_CASE["rename_payload_key_across_files"],
589+
True,
590+
),
591+
Fixture(
592+
"rename_payload_partial",
593+
"rename_payload_key_across_files",
594+
RENAME_PAYLOAD_PARTIAL,
595+
False,
596+
"format",
597+
),
598+
Fixture(
599+
"rename_payload_alias",
600+
"rename_payload_key_across_files",
601+
RENAME_PAYLOAD_ALIAS,
602+
False,
603+
"structure",
604+
),
605+
Fixture(
606+
"rename_payload_prefix",
607+
"rename_payload_key_across_files",
608+
RENAME_PAYLOAD_PREFIX,
609+
False,
610+
"behavior",
611+
),
582612
)

‎src/local_coding_slm/eval/cases_harder.py‎

Lines changed: 132 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,8 @@
22
33
These stay bounded (2–3 files). They isolate failure modes the seed and
44
extended corpus do not: leftover type aliases, catch-site drift, field
5-
aliases, a leftover keyword-parameter alias, and a public facade that
6-
must survive a helper signature change.
5+
aliases, a leftover keyword-parameter alias, a leftover payload-key
6+
alias, and a public facade that must survive a helper signature change.
77
88
Golden strings are fixtures for the scorer and stub Ollama, not live
99
model dumps. Live rates are not claimed here.
@@ -20,6 +20,7 @@
2020
"rename_dataclass_field",
2121
"widen_return_keep_facade",
2222
"rename_kwarg_across_files",
23+
"rename_payload_key_across_files",
2324
)
2425

2526

@@ -85,6 +86,12 @@ def _keyword_used(tree: ast.AST, name: str) -> bool:
8586
)
8687

8788

89+
def _string_used(tree: ast.AST, value: str) -> bool:
90+
return any(
91+
isinstance(node, ast.Constant) and node.value == value for node in ast.walk(tree)
92+
)
93+
94+
8895
def _function_returns_tuple(tree: ast.AST, name: str) -> bool:
8996
if not isinstance(tree, ast.Module):
9097
return False
@@ -762,6 +769,109 @@ def _kwarg_renamed(by_path: dict[str, str]) -> str | None:
762769
return None
763770

764771

772+
# --- rename payload key user_id -> account_id across builder and reader ---
773+
774+
RECORD_SOURCE = """\
775+
def make(name: str) -> dict[str, str]:
776+
return {"user_id": name}
777+
"""
778+
779+
LABEL_SOURCE = """\
780+
from record import make
781+
782+
783+
def label(name: str) -> str:
784+
payload = make(name)
785+
return f"acct:{payload['user_id']}"
786+
"""
787+
788+
RENAME_PAYLOAD_TASK = (
789+
"Rename the payload key user_id to account_id in record.py and label.py. "
790+
"Update make() so it writes account_id only, and label() so it reads "
791+
"account_id only. Do not keep user_id as a dict key, .get() fallback, "
792+
"or compatibility alias. Preserve label('ada') == 'acct:ada', "
793+
"label('') == 'acct:', and make('ada') == {'account_id': 'ada'}. "
794+
"Return two fenced Python files with path comments record.py and "
795+
"label.py, no prose."
796+
)
797+
798+
RENAME_PAYLOAD_GOLDEN = '''\
799+
```python
800+
# record.py
801+
def make(name: str) -> dict[str, str]:
802+
return {"account_id": name}
803+
```
804+
805+
```python
806+
# label.py
807+
from record import make
808+
809+
810+
def label(name: str) -> str:
811+
payload = make(name)
812+
return f"acct:{payload['account_id']}"
813+
```
814+
'''
815+
816+
RENAME_PAYLOAD_PARTIAL = '''\
817+
```python
818+
# record.py
819+
def make(name: str) -> dict[str, str]:
820+
return {"account_id": name}
821+
```
822+
'''
823+
824+
RENAME_PAYLOAD_ALIAS = '''\
825+
```python
826+
# record.py
827+
def make(name: str) -> dict[str, str]:
828+
return {"account_id": name, "user_id": name}
829+
```
830+
831+
```python
832+
# label.py
833+
from record import make
834+
835+
836+
def label(name: str) -> str:
837+
payload = make(name)
838+
return f"acct:{payload.get('account_id') or payload.get('user_id')}"
839+
```
840+
'''
841+
842+
RENAME_PAYLOAD_PREFIX = '''\
843+
```python
844+
# record.py
845+
def make(name: str) -> dict[str, str]:
846+
return {"account_id": name}
847+
```
848+
849+
```python
850+
# label.py
851+
from record import make
852+
853+
854+
def label(name: str) -> str:
855+
payload = make(name)
856+
return f"user:{payload['account_id']}"
857+
```
858+
'''
859+
860+
861+
def _payload_key_renamed(by_path: dict[str, str]) -> str | None:
862+
try:
863+
record = ast.parse(by_path.get("record.py", ""))
864+
label = ast.parse(by_path.get("label.py", ""))
865+
except SyntaxError as exc:
866+
return f"unparseable Python: {exc.msg}"
867+
for path, tree in (("record.py", record), ("label.py", label)):
868+
if _string_used(tree, "user_id") or _name_used(tree, "user_id"):
869+
return f"{path} still references user_id; do not keep a key alias"
870+
if not _string_used(tree, "account_id"):
871+
return f"{path} must use the account_id payload key"
872+
return None
873+
874+
765875
HARDER_CASES: tuple[EvalCase, ...] = (
766876
EvalCase(
767877
id="rename_exception_across_files",
@@ -834,18 +944,38 @@ def _kwarg_renamed(by_path: dict[str, str]) -> str | None:
834944
),
835945
max_tokens=1200,
836946
),
947+
EvalCase(
948+
id="rename_payload_key_across_files",
949+
tool="local_refactor",
950+
task=RENAME_PAYLOAD_TASK,
951+
files=(
952+
{"path": "record.py", "content": RECORD_SOURCE},
953+
{"path": "label.py", "content": LABEL_SOURCE},
954+
),
955+
required_paths=("record.py", "label.py"),
956+
required_top_level=("make", "label"),
957+
extra_structure=_payload_key_renamed,
958+
behavior=(
959+
BehaviorCheck("label", "label", ("ada",), "acct:ada"),
960+
BehaviorCheck("label", "label", ("",), "acct:"),
961+
BehaviorCheck("record", "make", ("ada",), {"account_id": "ada"}),
962+
),
963+
max_tokens=1200,
964+
),
837965
)
838966

839967
HARDER_GOLDEN = {
840968
"rename_exception_across_files": RENAME_EXCEPTION_GOLDEN,
841969
"rename_dataclass_field": RENAME_FIELD_GOLDEN,
842970
"widen_return_keep_facade": WIDEN_RETURN_GOLDEN,
843971
"rename_kwarg_across_files": RENAME_KWARG_GOLDEN,
972+
"rename_payload_key_across_files": RENAME_PAYLOAD_GOLDEN,
844973
}
845974

846975
HARDER_OBSERVED_FIRST = {
847976
"rename_exception_across_files": RENAME_EXCEPTION_PARTIAL,
848977
"rename_dataclass_field": RENAME_FIELD_PARTIAL,
849978
"widen_return_keep_facade": WIDEN_RETURN_PARTIAL,
850979
"rename_kwarg_across_files": RENAME_KWARG_PARTIAL,
980+
"rename_payload_key_across_files": RENAME_PAYLOAD_PARTIAL,
851981
}

‎tests/test_eval_harder.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ def _fixture(name: str):
2121

2222
class HarderMultifileCorpusTests(unittest.TestCase):
2323
def test_harder_cases_span_more_than_one_file(self) -> None:
24-
self.assertEqual(len(HARDER_CASE_IDS), 4)
24+
self.assertEqual(len(HARDER_CASE_IDS), 5)
2525
for case_id in HARDER_CASE_IDS:
2626
with self.subTest(case_id=case_id):
2727
case = CASES_BY_ID[case_id]
@@ -35,6 +35,7 @@ def test_goldens_pass_all_four_layers(self) -> None:
3535
"rename_field_golden",
3636
"widen_return_golden",
3737
"rename_kwarg_golden",
38+
"rename_payload_golden",
3839
):
3940
with self.subTest(fixture=name):
4041
fixture = _fixture(name)
@@ -49,6 +50,7 @@ def test_alias_and_int_return_are_structure_not_behavior(self) -> None:
4950
"rename_field_alias",
5051
"widen_return_int",
5152
"rename_kwarg_alias",
53+
"rename_payload_alias",
5254
):
5355
with self.subTest(fixture=name):
5456
fixture = _fixture(name)
@@ -63,6 +65,7 @@ def test_threshold_format_and_rate_are_behavior(self) -> None:
6365
"rename_field_format",
6466
"widen_return_wrong_rate",
6567
"rename_kwarg_separator",
68+
"rename_payload_prefix",
6669
):
6770
with self.subTest(fixture=name):
6871
fixture = _fixture(name)
@@ -95,6 +98,7 @@ def test_partial_files_are_format(self) -> None:
9598
"rename_field_partial",
9699
"widen_return_partial",
97100
"rename_kwarg_partial",
101+
"rename_payload_partial",
98102
):
99103
with self.subTest(fixture=name):
100104
fixture = _fixture(name)

0 commit comments

Comments
 (0)