evals: persist judge model usage on the legacy G-Eval path by tawnymanticore · Pull Request #1629 · Kiln-AI/Kiln

tawnymanticore · 2026-07-28T16:05:59Z

Rescoped. scosman/evals_v2 has since grown judge-usage capture for V2 evals on its own. This branch is now only the one lane the base does not cover: the legacy G-Eval / LLM-as-judge path. 12 files → 6, 308 insertions → 127.

What the base already does

Everything this PR originally added for V2 evals is on scosman/evals_v2 today, and is dropped here in favour of the base's version:

EvalRun.eval_usage and V2EvalResult.usage in the datamodel.
LlmJudgeEval.evaluate keeping the judge TaskRun's usage.
_persist_score / _persist_judgment writing it on every V2 record.
A datamodel default-None + round-trip test (test_eval_usage_defaults_to_none_and_round_trips), so this branch's duplicate is gone.

What is left

EvalRunner.run_job dispatches on eval_config.config_type: v2 goes to _run_v2_job, and g_eval / llm_as_judge go to _run_legacy_job. The legacy path still throws the judge run away, so no g_eval-configured eval on disk records what its own scoring cost.

GEval.run_eval keeps the judge TaskRun instead of dropping it (_, run_output = ...) and returns its usage. The value covers the full judgment — the adapter sums usage across every model call in the invocation, so the two-call COT heuristic is aggregated, not just the last call.
BaseEval.run_eval / run_task_and_eval carry it through as an extra tuple element; BaseV2EvalBridge forwards the base's V2EvalResult.usage.
_run_legacy_job sets eval_usage on the EvalRun it saves.

Non-LLM evals persist None — honestly unset rather than zero.

Why not just close it

The legacy path is live, not sunset: legacy_eval_adapter_from_type still returns GEval for both g_eval and llm_as_judge, and any eval config already on disk with those types still runs through it. Until that dispatch arm is deleted, this is the last hole in per-lane usage accounting. If legacy evals are being removed outright, closing this is a one-click alternative — nothing else in the branch survives on its own.

Back-compat

Unchanged from the original: eval_usage is an optional field defaulting to None, already on the base. The only behaviour change is that legacy-judge records written after this carry a value where they previously carried None.

Testing

libs/core/kiln_ai/adapters/eval/ + test_eval_model.py + test_eval_api.py: 1090 passed.
New: test_run_job_persists_judge_eval_usage — a judged legacy EvalRun carries the judge's usage, through run_job and a disk round-trip.
Updated: evaluator mocks and stubs widened to the new tuple arity, including the base's own test_run_job_full_trace_serializes_per_message_usage.
Full libs/ + app/desktop run: 7507 passed. The 5 failures and 5 errors (vector-store, chunker, model-cache benchmark, document-api, and tkinter-less desktop imports) all reproduce on origin/scosman/evals_v2 with this branch's changes absent.
ruff format --check clean on every touched file.

Judging spend is currently unmeasurable: the judge call's usage is assembled by the adapter (aggregated across every LLM call the judgment makes, including the two-call COT heuristic) and then discarded with the un-saved judge TaskRun. - EvalRun gains eval_usage: Usage | None — the judge model's usage, distinct from task_run_usage (the evaluated run's usage). Default None keeps every existing record loading unchanged. - GEval.run_eval and the V2 LlmJudgeEval keep the judge TaskRun's usage; BaseEval.run_eval/run_task_and_eval return it alongside the scores, and V2EvalResult carries it as eval_usage. - eval_runner threads it onto every EvalRun construction that ran an evaluator (legacy path + the V2 EvalInput / task_run_eval / eval_config_eval paths). Non-LLM evals persist None. - api_schema.d.ts regenerated for the new field. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gemini-code-assist · 2026-07-28T16:06:02Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

coderabbitai · 2026-07-28T16:06:24Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 03322906-ae15-4768-aa00-fa3663dcf271

📥 Commits

Reviewing files that changed from the base of the PR and between 8b8fa06 and cb19043.

📒 Files selected for processing (4)

libs/core/kiln_ai/adapters/eval/base_eval.py
libs/core/kiln_ai/adapters/eval/eval_runner.py
libs/core/kiln_ai/adapters/eval/test_base_eval.py
libs/core/kiln_ai/adapters/eval/test_eval_runner.py

💤 Files with no reviewable changes (1)

libs/core/kiln_ai/adapters/eval/test_eval_runner.py

🚧 Files skipped from review as they are similar to previous changes (2)

libs/core/kiln_ai/adapters/eval/base_eval.py
libs/core/kiln_ai/adapters/eval/test_base_eval.py

Included review availability: Your plan provides up to 8 included reviews per hour; 5 remain after this review.

Walkthrough

The evaluation pipeline now returns optional model usage metadata from evaluators, propagates it through task execution, and stores it on legacy EvalRun records. Tests cover judge usage, absent usage, and disk persistence.

Changes

Evaluation usage tracking

Layer / File(s)	Summary
Usage data contracts `libs/core/kiln_ai/adapters/eval/base_eval.py`, `libs/core/kiln_ai/adapters/eval/test_base_eval.py`	Evaluation methods and test adapters now return an optional `Usage` value. The V2 bridge returns `result.usage` and preserves skipped-evaluation errors.
Judge usage capture `libs/core/kiln_ai/adapters/eval/g_eval.py`, `libs/core/kiln_ai/adapters/eval/test_g_eval.py`	`GEval` returns usage from the judge `TaskRun` in both evaluation modes. Tests unpack and validate the additional value.
Eval run persistence `libs/core/kiln_ai/adapters/eval/eval_runner.py`, `libs/core/kiln_ai/adapters/eval/test_eval_runner.py`	The legacy runner stores evaluator usage on `EvalRun`. Tests cover V1 usage persistence, disk reloads, and updated evaluator result tuples.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: 🔵 Low · up to cb190

The change is localized to evaluation-usage persistence with no supplied correctness or deployment failures; it is mergeable with explicit owner awareness that the requested Python validation, including type checking, should be confirmed.

Sequence Diagram(s)

sequenceDiagram
  participant GEval
  participant JudgeAdapter
  participant BaseEval
  participant EvalRunner
  participant EvalRun
  GEval->>JudgeAdapter: invoke judge evaluation
  JudgeAdapter-->>GEval: return TaskRun with usage
  GEval-->>BaseEval: return scores, outputs, and usage
  BaseEval-->>EvalRunner: return task result and eval usage
  EvalRunner->>EvalRun: persist eval_usage

Possibly related PRs

Kiln-AI/Kiln#391: Modifies the eval pipeline to propagate and persist Usage metadata.
Kiln-AI/Kiln#1645: Extends eval_runner.py with structured usage persistence from another evaluation source.
Kiln-AI/Kiln#1683: Propagates evaluator usage into EvalRun through overlapping evaluation files.

Suggested reviewers: sfierro, claude, scosman

Poem

A rabbit tracks each judge’s count,
Through scores and runs the tokens mount.
The usage hops from task to store,
Then disk remembers even more.
“No missing tuple!” the rabbit sings.
🐇 Metrics sprout on evaluation wings.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 22.06% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The description explains the implementation and testing, but it omits the required Related Issues, Contributor License Agreement, and Checklists sections.	Add the required Related Issues, Contributor License Agreement confirmation, and Checklists sections with applicable items completed.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main change: persisting judge model usage for the legacy G-Eval path.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch fix/evalrun-judge-usage

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-28T16:09:14Z

📊 Coverage Report

Overall Coverage: 92%

Diff: origin/scosman/evals_v2...HEAD

libs/core/kiln_ai/adapters/eval/base_eval.py (75.0%): Missing lines 485
libs/core/kiln_ai/adapters/eval/eval_runner.py (100%)
libs/core/kiln_ai/adapters/eval/g_eval.py (25.0%): Missing lines 330,335,341

Summary

Total: 10 lines
Missing: 4 lines
Coverage: 60%

Line-by-line

View line-by-line diff coverage

libs/core/kiln_ai/adapters/eval/base_eval.py

Lines 481-486

  481         if result.skipped_reason is not None:
  482             raise ValueError(
  483                 f"V2 eval was skipped ({result.skipped_reason}): {result.skipped_detail}"
  484             )
! 485         return result.scores, result.intermediate_outputs, result.usage

libs/core/kiln_ai/adapters/eval/g_eval.py

Lines 326-339

  326 
  327         # The judge TaskRun is never persisted (allow_saving=False), but its usage covers
  328         # every LLM call the judgment made (the COT heuristic can make two), so we keep it.
  329         # invoke_returning_run_output() also runs validations for us over _run().
! 330         judge_run, run_output = await adapter.invoke_returning_run_output(
  331             run_description
  332         )
  333 
  334         if self.eval_config.config_type == EvalConfigType.llm_as_judge:
! 335             return (
  336                 self.build_llm_as_judge_score(run_output),
  337                 run_output.intermediate_outputs,
  338                 judge_run.usage,
  339             )

Lines 337-345

  337                 run_output.intermediate_outputs,
  338                 judge_run.usage,
  339             )
  340         else:
! 341             return (
  342                 self.build_g_eval_score(run_output),
  343                 run_output.intermediate_outputs,
  344                 judge_run.usage,
  345             )

📊 HTML Coverage Report - Interactive coverage report
📈 Diff Coverage Report - Detailed diff analysis
Github Actions Run - View the full coverage report

The base now records judge usage for V2 evals on its own: `EvalRun.eval_usage` and `V2EvalResult.usage` are in the datamodel, `LlmJudgeEval.evaluate` captures the judge TaskRun's usage, and `_persist_score` / `_persist_judgment` write it on every V2 record. All of that is dropped here in favour of the base's version — including the duplicate datamodel round-trip test, which the base already has as test_eval_usage_defaults_to_none_and_round_trips. What is left is the one lane the base does not cover: `_run_legacy_job`, which still discards the judge run for g_eval / llm_as_judge configs. That is: - GEval.run_eval keeps the judge TaskRun instead of dropping it on the floor, and returns its usage. The value covers the whole judgment — the adapter sums usage across every call, so the two-call COT heuristic is aggregated. - BaseEval.run_eval / run_task_and_eval carry it through as a third/fourth tuple element; BaseV2EvalBridge forwards the base's V2EvalResult.usage. - _run_legacy_job sets eval_usage on the EvalRun it saves. Non-LLM evals persist None — honestly unset rather than zero. libs/core/kiln_ai/adapters/eval/ + test_eval_model.py + test_eval_api.py: 1090 passed. Full libs/ + app/desktop run: 7507 passed, with 5 failures and 5 errors that all reproduce on the base (vector-store, chunker, model-cache benchmark, document-api, and tkinter-less desktop imports).

tawnymanticore · 2026-08-18T22:07:02Z

Closing — legacy evals are being sunset, and the legacy G-Eval path was all this branch had left.

For the record of what was verified before closing: everything else this PR originally added has independently landed on scosman/evals_v2 — EvalRun.eval_usage, V2EvalResult.usage, the judge-usage capture in LlmJudgeEval.evaluate, and the wiring through _persist_score / _persist_judgment. Judge usage on the V2 lane is fully covered there, including a datamodel round-trip test. Nothing is lost by closing this.

The only thing that goes with it is _run_legacy_job recording eval_usage for g_eval / llm_as_judge configs. Those still run today (legacy_eval_adapter_from_type returns GEval for both, and run_job routes to _run_legacy_job when config_type != v2), so until that dispatch arm is deleted, legacy-judge EvalRuns will keep persisting eval_usage=None. That is the accepted cost of the sunset — the field is optional and None already means "not measured".

If legacy evals turn out to have a longer tail than expected, the change is ~15 lines of production code: keep the judge TaskRun in GEval.run_eval instead of discarding it, widen the BaseEval.run_eval / run_task_and_eval tuples, and set eval_usage= on the EvalRun in _run_legacy_job. The branch fix/evalrun-judge-usage is left in place at cb19043c (all CI green) if it's ever wanted back.

Generated by Claude Code

coderabbitai Bot mentioned this pull request Jul 30, 2026

Fix V2 evals crashing on every dataset item with dataset_id validation error #1632

Merged

2 tasks

tawnymanticore requested a review from sfierro August 4, 2026 13:41

tawnymanticore mentioned this pull request Aug 4, 2026

evals: record the synthetic user's tokens, not just its cost #1645

Merged

coderabbitai Bot mentioned this pull request Aug 15, 2026

Migrate eval runs from inline traces to trace pointers #1676

Merged

tawnymanticore changed the title ~~evals: persist judge model usage on EvalRun (eval_usage)~~ evals: persist judge model usage on the legacy G-Eval path Aug 18, 2026

tawnymanticore closed this Aug 18, 2026

Navigation Menu

evals: persist judge model usage on the legacy G-Eval path - #1629

evals: persist judge model usage on the legacy G-Eval path#1629
tawnymanticore wants to merge 2 commits into
scosman/evals_v2from
fix/evalrun-judge-usage

tawnymanticore commented Jul 28, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jul 28, 2026

Uh oh!

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading

❌ Failed checks (2 warnings)

Uh oh!

github-actions Bot commented Jul 28, 2026 •

edited

Loading

libs/core/kiln_ai/adapters/eval/base_eval.py

libs/core/kiln_ai/adapters/eval/g_eval.py

Uh oh!

tawnymanticore commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tawnymanticore commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What the base already does

What is left

Why not just close it

Back-compat

Testing

Uh oh!

gemini-code-assist Bot commented Jul 28, 2026

Uh oh!

coderabbitai Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (2 warnings)

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Coverage Report

Diff: origin/scosman/evals_v2...HEAD

Summary

Line-by-line

libs/core/kiln_ai/adapters/eval/base_eval.py

libs/core/kiln_ai/adapters/eval/g_eval.py

Uh oh!

tawnymanticore commented Aug 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tawnymanticore commented Jul 28, 2026 •

edited

Loading

coderabbitai Bot commented Jul 28, 2026 •

edited

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading