Add val_set_filter_id to Eval: three-way train/val/test dataset splits by scosman · Pull Request #1620 · Kiln-AI/Kiln

scosman · 2026-07-23T05:58:29Z

What does this PR do?

Adds a validation split to evals, so run methods can be compared on held-out data without touching the test set. Evals previously carried only two split filters: train_set_filter_id and eval_set_filter_id (which is the test set — the name is legacy).

Add Eval.val_set_filter_id with the same lazy migration as train: evals loaded without one get tag::val_{name_slug}
Document on eval_set_filter_id that it is the test set and the name is legacy
Mint val_{name_slug} tags/filter ids in spec eval creation (generate_spec_eval_tags / generate_spec_eval_filter_ids and both callers: spec_api and copilot_api)
Deal a val split in copilot synthetic dataset creation: eval keeps half of the post-golden pool; the other half now splits 2/3 train, 1/3 val
Regenerate the web client schema for the new Eval field

Related Issues

None — approved eval-API improvement (val split groundwork).

Checklists

Tests have been run locally and passed
New tests have been added to any work in /lib

🤖 Generated with Claude Code

https://claude.ai/code/session_0135zWTBh8MRPwCXiWvuE597

Generated by Claude Code

Evals previously carried only two split filters: train_set_filter_id and eval_set_filter_id (which is the test set - the name is legacy). This adds a validation split so run methods can be compared on held-out data without touching the test set. - Add Eval.val_set_filter_id with the same lazy migration as train: evals loaded without one get tag::val_{name_slug} - Document on eval_set_filter_id that it is the test set and the name is legacy - Mint val_{name_slug} tags/filter ids in spec eval creation (generate_spec_eval_tags / generate_spec_eval_filter_ids and both callers: spec_api and copilot_api) - Deal a val split in copilot synthetic dataset creation: eval keeps half of the post-golden pool; the other half now splits 2/3 train, 1/3 val - Regenerate the web client schema for the new Eval field

gemini-code-assist · 2026-07-23T05:58:32Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

cursor · 2026-07-23T05:58:34Z

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

coderabbitai · 2026-07-23T05:58:49Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a30860e4-873f-4e18-a5b3-a85fa19938a6

📥 Commits

Reviewing files that changed from the base of the PR and between bc9edc9 and 5e66282.

📒 Files selected for processing (11)

app/desktop/studio_server/copilot_api.py
app/desktop/studio_server/test_copilot_api.py
app/desktop/studio_server/utils/copilot_utils.py
app/desktop/studio_server/utils/test_copilot_utils.py
app/web_ui/src/lib/api_schema.d.ts
libs/core/kiln_ai/datamodel/eval.py
libs/core/kiln_ai/datamodel/test_eval_model.py
libs/server/kiln_server/spec_api.py
libs/server/kiln_server/test_spec_api.py
libs/server/kiln_server/utils/spec_utils.py
libs/server/kiln_server/utils/test_spec_utils.py

Walkthrough

Eval creation and dataset task-run splitting now include a validation dataset. Tag and filter-ID generation, persistence schemas, legacy migrations, server creation, desktop Copilot wiring, and tests were updated accordingly.

Changes

Validation Dataset Support

Layer / File(s)	Summary
Eval validation filter model and migration `libs/core/kiln_ai/datamodel/eval.py`, `libs/core/kiln_ai/datamodel/test_eval_model.py`, `app/web_ui/src/lib/api_schema.d.ts`	`Eval` defines `val_set_filter_id`, documents filter exclusivity, and generates missing validation filter IDs when loading legacy evals.
Server spec validation tag wiring `libs/server/kiln_server/utils/spec_utils.py`, `libs/server/kiln_server/spec_api.py`, `libs/server/kiln_server/utils/test_spec_utils.py`, `libs/server/kiln_server/test_spec_api.py`	Spec tag and filter-ID helpers produce validation values, and `create_spec` stores the validation filter ID on the generated eval.
Desktop dataset split and task-run wiring `app/desktop/studio_server/utils/copilot_utils.py`, `app/desktop/studio_server/copilot_api.py`, `app/desktop/studio_server/utils/test_copilot_utils.py`, `app/desktop/studio_server/test_copilot_api.py`	Copilot dataset creation passes validation tags and allocates validation task runs from the remaining examples.
Estimated code review effort: 3 (Moderate)	~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant create_spec
  participant generate_spec_eval_tags
  participant generate_spec_eval_filter_ids
  participant Eval
  create_spec->>generate_spec_eval_tags: generate eval, train, val, and golden tags
  create_spec->>generate_spec_eval_filter_ids: generate four dataset filter IDs
  create_spec->>Eval: construct Eval with val_set_filter_id

Suggested reviewers: sfierro

Poem

A bunny sorts the datasets bright,
Eval, train, val, golden in flight.
Tags hop neatly, filters align,
Legacy evals migrate fine.
“Four little splits!” we cheer tonight.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 62.96% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding validation-set filtering and three-way dataset splits for evals.
Description check	✅ Passed	The description matches the required template and includes the goal, related issues, CLA confirmation, and checklist items.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch scosman/val-set

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-07-23T06:01:08Z

📊 Coverage Report

Overall Coverage: 92%

Diff: origin/main...HEAD

app/desktop/studio_server/copilot_api.py (100%)
app/desktop/studio_server/utils/copilot_utils.py (100%)
libs/core/kiln_ai/datamodel/eval.py (92.9%): Missing lines 479
libs/server/kiln_server/spec_api.py (100%)
libs/server/kiln_server/utils/spec_utils.py (100%)

Summary

Total: 28 lines
Missing: 1 line
Coverage: 96%

Line-by-line

View line-by-line diff coverage

libs/core/kiln_ai/datamodel/eval.py

Lines 475-483

  475         Generates a tag-based filter ID from the eval name following the convention
  476         used by spec-based evals (e.g., "val_{name_slug}").
  477         """
  478         if self.id is None:
! 479             return self
  480 
  481         if not self._loaded_from_file:
  482             return self

📊 HTML Coverage Report - Interactive coverage report
📈 Diff Coverage Report - Detailed diff analysis
Github Actions Run - View the full coverage report

This was referenced Jul 23, 2026

Add optional split param to the eval APIs: run or read results by train/val/test #1621

Closed

Add per-score direction flag to eval output scores #1622

Open

coderabbitai Bot mentioned this pull request Jul 29, 2026

KIL-741: Ensure non-eval-builder eval creation works for all cases #1630

Closed

This was referenced Aug 6, 2026

(do not merge): eval splits in the eval's own item source, not only TaskRuns #1657

Closed

Train/val/test splits across V1 (TaskRun) and V2 (EvalInput) eval data sources #1662

Merged

coderabbitai Bot mentioned this pull request Aug 17, 2026

Merge scosman/evals_v2 into sfierro/KIL-741 #1683

Merged

2 tasks

scosman closed this Aug 19, 2026

coderabbitai Bot mentioned this pull request Aug 19, 2026

Rename eval_tag to test_tag throughout codebase #1698

Merged

2 tasks

Navigation Menu

Uh oh!

Add val_set_filter_id to Eval: three-way train/val/test dataset splits - #1620

Add val_set_filter_id to Eval: three-way train/val/test dataset splits#1620
scosman wants to merge 1 commit into
mainfrom
scosman/val-set

scosman commented Jul 23, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jul 23, 2026

Uh oh!

cursor Bot commented Jul 23, 2026

Uh oh!

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jul 23, 2026

libs/core/kiln_ai/datamodel/eval.py

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

scosman commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Related Issues

Checklists

Uh oh!

gemini-code-assist Bot commented Jul 23, 2026

Uh oh!

cursor Bot commented Jul 23, 2026

Uh oh!

coderabbitai Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Jul 23, 2026

📊 Coverage Report

Diff: origin/main...HEAD

Summary

Line-by-line

libs/core/kiln_ai/datamodel/eval.py

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

scosman commented Jul 23, 2026 •

edited

Loading

coderabbitai Bot commented Jul 23, 2026 •

edited

Loading