Pin min-threads and env-var ground truth in migrate-dotnet8-to-dotnet9 rubric

[AbhitejJohn]

What

Strengthens two stimulus rubrics in migrate-dotnet8-to-dotnet9/eval.yaml so the judge has explicit ground truth for the .NET 8 -> 9 runtime-configuration change.

In .NET 9, environment variables take precedence over runtimeconfig.json for GC and ThreadPool settings. As a result:

• server GC becomes workstation GC,
• concurrent GC is disabled,
• the minimum ThreadPool thread count drops from 25 to 4.

The rubric now states that a response claiming any of these values are unchanged (for example, that min threads stay at 25) is factually wrong however confidently it is stated.

Why

In a prior dual-judge run on this exact skill, the two judges split on the "env var precedence reversal" stimulus: the primary judge scored the baseline answer as correct while the second judge scored the treatment answer as correct. The reversal is a factual matter with one right answer. Pinning the ground truth in the rubric gives the judge the fact it needs, which should reduce that kind of split.

Design note: rubric-only, on purpose

Deterministic output-matches / output-not-matches graders were tried and rejected. A regex cannot distinguish a correct answer that mentions the old value in a negated or past-tense clause ("no longer 25", "stayed at 25 in .NET 8") from an incorrect answer, so graders would false-fail correct text on both arms and corrupt the very eval we want to make reliable. The rubric guides the judge without ever hard-failing a valid answer.

Scope / blast radius

• One file, two rubric lines. No grader changes. No cross-skill effect.
• eng/eval-quality/check_eval_quality.py exits 0.

Verification still needed

The live judge cannot be run locally. A live eval run is needed to confirm the strengthened rubric actually changes the verdict on these stimuli.

/cc @AbhitejJohn

[github-actions]

Skill Coverage Report

	Plugin	Skill	Covered	Coverage
✅	dotnet-upgrade	migrate-dotnet8-to-dotnet9	6/6	100%

[AbhitejJohn]

/evaluate 30b0125

[github-actions]

📊 Skill Evaluation Results

2 model/skill results across 1 skill and 2 models — ✅ 1 improved, ➖ 1 not proven improved, ⚠️ 0 invalid or underpowered, 📉 0 preference losses (report only).

Measurement identity: evaluated commit 30b01253ce2186b0cec5a3a0c32eb7393a8689ee; 2 judge models.

Measurement health: 2 expected / 2 observed / 2 written; 0 missing, 0 unexpected, 0 invalid; 0 recovered comparison error slots and 0 unresolved comparison error slots.

Objective completion gate: not enabled. Aggregate completion transitions are telemetry only, so this report does not claim that zero objective regressions were proven.

A result passes only when the aggregate net win across distinct-stimulus votes is at least 20% and an exact one-sided sign-test result of p ≤ 0.05. Repeated runs measure reliability only.

Skill	Model	Verdict	Gate evidence	Overfit	Warnings	Next action
migrate-dotnet8-to-dotnet9	claude-sonnet-4.6	✅ Improved	n=12; 10W/2T/0L; d=10; p=0.001; net +83.3%	✅ 0.19	—	None.
migrate-dotnet8-to-dotnet9	gpt-5.6-luna	➖ Not proven improved	n=12; 5W/1T/6L; d=11; p=0.500; net -8.3%	✅ 0.06	—	Inspect tied or lost stimuli and fix inconsistent skill behavior.

ℹ️ How to read this report

• ✅ Improved — the result passed both the statistical gate and the 20% practical net-win floor.
• ➖ Not proven improved — the result is valid but did not pass both gates. This is not automatically a regression.
• ⚠️ Invalid / underpowered — the gate withheld a quality verdict. Fix the measurement before judging the skill.
• 📉 Preference loss — the LLM judge credibly preferred baseline. It is report-only, not objective completion proof.
• Gate evidence — n distinct-stimulus votes, W/T/L stimulus votes, d discordant votes, exact one-sided p, and net win. The p value applies to one model/skill result; no matrix-wide multiple-comparison correction is applied.
• Overfit — overfitting-judge severity (✅ Low, 🟡 Moderate, 🔴 High, — none) and score.
• Warnings — activation, timeout, retry recovery, or unresolved comparison conditions that need attention.
• Do not add repeated runs to increase statistical power. Do not add stimuli after seeing a near-pass unless the new breadth is predeclared for a new experiment.

➖ Not proven improved — migrate-dotnet8-to-dotnet9 (gpt-5.6-luna)

Why: Net win -8.3% (5W/1T/6L over 12 stimulus vote(s), sign test p=0.500), mean preference -13.3% across 12 paired run(s) — no improvement

Next action: Inspect tied or lost stimuli and fix inconsistent skill behavior.

State: VALID_NO_CHANGE (no_credible_preference_change)

Gate evidence: n=12; 5W/1T/6L; d=11; p=0.500; net -8.3%

Overfit: Low (score 0.06)

Repeated-run reliability (not used by the gate): 12 paired runs (5W/1T/6L).

Weak or warning scenarios:

Scenario	Net win	Δ Pref	Runs (W/T/L)
▼ App with JsonDocument null deserialization and BinaryFormatter fallback	-100.0%	-100.0%	0/0/1
= C# 13 compiler breaking changes — InlineArray on record, iterator safe context, collection expressions	+0.0%	+0.0%	0/1/0
▼ Containerized app with zlib dependency and runtime configuration	-100.0%	-40.0%	0/0/1
▼ EF Core Cosmos DB app with discriminator and sync I/O	-100.0%	-100.0%	0/0/1
▼ EF Core Cosmos DB app with existing documents and composite id format	-100.0%	-100.0%	0/0/1
▼ EF Core app with migration patterns and Cosmos DB discriminator	-100.0%	-40.0%	0/0/1
▼ Library with String.Trim span overload, keyed services, and InlineArray	-100.0%	-40.0%	0/0/1

Illustrative judge evidence:

• App with JsonDocument null deserialization and BinaryFormatter fallback: The task was to migrate to .NET 9 AND fix all breaking changes. Response A completed the migration: updated TargetFramework to net9.0, replaced BinaryFormatter with System.Text.Json, added missing usings, and made UserAgent null-safe, achieving a clean build. However, A's appr...

This is one example, not the aggregate verdict. Open Full Results for every judgment.

Routine passing details for 1 result are in Full Results.

🔍 Full Results - all metrics and investigation details

To investigate non-passing or warning results, paste this to your AI coding agent:

For PR 1046 in dotnet/skills, download eval artifacts with gh run download 32816309234 --repo dotnet/skills --pattern "vally-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/30b01253ce2186b0cec5a3a0c32eb7393a8689ee/eng/vally-adapter/InvestigatingResults.md and follow it. Classify each result as measurement-invalid, underpowered, not-proven, preference-loss, or passing-with-warning. Use stateReason, result accounting, weak scenarios, and judge evidence to give the cause and exact next fix.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions