Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ jobs:
python3 -m pip install jsonschema
python3 eval/runners/test_portability.py
- name: Check matcher validation
run: python3 eval/runners/test_match.py
run: |
python3 eval/runners/test_match.py
python3 eval/runners/test_gold_judge.py
- name: Check internal links
uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
with:
Expand Down
10 changes: 10 additions & 0 deletions docs/evals.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,16 @@ artifacts document case by case:

## Runs (audit tier)

The first sweep's 15 zero-correctness-recall CRB cases were adjudicated on
2026-10-02: 35 labels became 9 eligible correctness defects, 17
policy-excluded findings, and 9 wrong or insufficiently specified labels.
The unchanged historical candidate already matched one of the 9; its
map-race label had been incorrectly tiered as efficiency. See the
[rulings and historical re-judge](../eval/runs/2026-10-02-zero-recall-adjudication/README.md).
This selected cluster is not a new full-corpus recall result. The new
[30-pair Codex calibration](../eval/calibration/README.md) does not validate
the archived Claude match decisions.

The deep tier's first recorded measurement. Not a gold row: `reviso-6`
carries no labels file, so there is no recall figure to report — what this
run establishes is the score distribution and the per-lens yield, which
Expand Down
240 changes: 240 additions & 0 deletions eval/calibration/adjudication-2026-10-02.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
{
"date": "2026-10-02",
"cases": [
{
"case": "crb-grafana-76186",
"labels": "eval/corpus/labels/crb-grafana-76186.json",
"dispositions": [
{
"finding_index": 0,
"status": "policy-excluded"
},
{
"finding_index": 1,
"status": "label-wrong"
}
]
},
{
"case": "crb-grafana-79265",
"labels": "eval/corpus/labels/crb-grafana-79265.json",
"dispositions": [
{
"finding_index": 0,
"status": "real-miss"
},
{
"finding_index": 1,
"status": "policy-excluded"
},
{
"finding_index": 2,
"status": "label-wrong"
},
{
"finding_index": 3,
"status": "policy-excluded"
},
{
"finding_index": 4,
"status": "label-wrong"
}
]
},
{
"case": "crb-grafana-90939",
"labels": "eval/corpus/labels/crb-grafana-90939.json",
"dispositions": [
{
"finding_index": 0,
"status": "policy-excluded"
},
{
"finding_index": 1,
"status": "policy-excluded"
}
]
},
{
"case": "crb-grafana-97529",
"labels": "eval/corpus/labels/crb-grafana-97529.json",
"dispositions": [
{
"finding_index": 0,
"status": "policy-excluded"
},
{
"finding_index": 1,
"status": "real-miss"
}
]
},
{
"case": "crb-grafana-103633",
"labels": "eval/corpus/labels/crb-grafana-103633.json",
"dispositions": [
{
"finding_index": 0,
"status": "policy-excluded"
},
{
"finding_index": 1,
"status": "policy-excluded"
}
]
},
{
"case": "crb-grafana-107534",
"labels": "eval/corpus/labels/crb-grafana-107534.json",
"dispositions": [
{
"finding_index": 0,
"status": "policy-excluded"
}
]
},
{
"case": "crb-keycloak-36880",
"labels": "eval/corpus/labels/crb-keycloak-36880.json",
"dispositions": [
{
"finding_index": 0,
"status": "real-miss"
},
{
"finding_index": 1,
"status": "label-wrong"
},
{
"finding_index": 2,
"status": "policy-excluded"
}
]
},
{
"case": "crb-keycloak-36882",
"labels": "eval/corpus/labels/crb-keycloak-36882.json",
"dispositions": [
{
"finding_index": 0,
"status": "policy-excluded"
}
]
},
{
"case": "crb-keycloak-38446",
"labels": "eval/corpus/labels/crb-keycloak-38446.json",
"dispositions": [
{
"finding_index": 0,
"status": "policy-excluded"
},
{
"finding_index": 1,
"status": "real-miss"
}
]
},
{
"case": "crb-sentry-80528",
"labels": "eval/corpus/labels/crb-sentry-80528.json",
"dispositions": [
{
"finding_index": 0,
"status": "policy-excluded"
},
{
"finding_index": 1,
"status": "policy-excluded"
}
]
},
{
"case": "crb-sentry-95633",
"labels": "eval/corpus/labels/crb-sentry-95633.json",
"dispositions": [
{
"finding_index": 0,
"status": "label-wrong"
},
{
"finding_index": 1,
"status": "policy-excluded"
},
{
"finding_index": 2,
"status": "policy-excluded"
}
]
},
{
"case": "crb-sentry-greptile-3",
"labels": "eval/corpus/labels/crb-sentry-greptile-3.json",
"dispositions": [
{
"finding_index": 0,
"status": "real-miss"
},
{
"finding_index": 1,
"status": "label-wrong"
},
{
"finding_index": 2,
"status": "real-miss"
}
]
},
{
"case": "crb-sentry-greptile-5",
"labels": "eval/corpus/labels/crb-sentry-greptile-5.json",
"dispositions": [
{
"finding_index": 0,
"status": "label-wrong"
},
{
"finding_index": 1,
"status": "policy-excluded"
},
{
"finding_index": 2,
"status": "real-miss"
}
]
},
{
"case": "crb-cal.com-14943",
"labels": "eval/corpus/labels/crb-cal.com-14943.json",
"dispositions": [
{
"finding_index": 0,
"status": "real-miss"
},
{
"finding_index": 1,
"status": "real-miss"
}
]
},
{
"case": "crb-discourse-graphite-9",
"labels": "eval/corpus/labels/crb-discourse-graphite-9.json",
"dispositions": [
{
"finding_index": 0,
"status": "label-wrong"
},
{
"finding_index": 1,
"status": "label-wrong"
}
]
}
],
"counts": {
"policy-excluded": 17,
"label-wrong": 9,
"real-miss": 9
}
}
17 changes: 17 additions & 0 deletions eval/corpus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,23 @@ tiers are not measured against the `slop-*` cases.

## Imported gold cases (CRB)

The 15 zero-correctness-recall cases from the first sweep were adjudicated
on 2026-10-02. Their finding text remains intact; `adjudication` records a
disposition, reason, and pinned source evidence. `policy-excluded` and
`label-wrong` findings are excluded from gold matching and the correctness
denominator, but retained in the judge's `excluded_gold` array. A case whose
gold is entirely excluded is **unscored**, not expected-clean: unmatched
candidate findings remain promotion candidates, not proven false positives.
One imported category was corrected explicitly: Grafana's unlocked cache
iteration can panic, so it is correctness rather than efficiency; the
original category is retained in `imported_category`.

The [adjudication manifest](../calibration/adjudication-2026-10-02.json)
indexes all 35 rulings. The
[historical re-judge](../runs/2026-10-02-zero-recall-adjudication/summary.json)
uses the original candidate outputs and original match indices; it does
not retroactively rerun the old model or claim current-release recall.

50 entries came from `withmartian/code-review-benchmark` via
`import-crb.sh <fixtures-dir>` (re-run is idempotent): real PRs in
calcom/cal.com (ts), grafana/grafana (go), keycloak/keycloak (java),
Expand Down
24 changes: 22 additions & 2 deletions eval/corpus/labels/crb-cal.com-14943.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,17 @@
"severity": "high",
"category": "correctness",
"crb_category": "bug",
"description": "Using retryCount: reminder.retryCount + 1 reads a possibly stale value and can lose increments under concurrency; consider an atomic increment via Prisma (increment: 1) to avoid race conditions (also applies to the similar update in the catch block)."
"description": "Using retryCount: reminder.retryCount + 1 reads a possibly stale value and can lose increments under concurrency; consider an atomic increment via Prisma (increment: 1) to avoid race conditions (also applies to the similar update in the catch block).",
"adjudication": {
"date": "2026-10-02",
"status": "real-miss",
"reason": "concurrency",
"why": "The new retry updates assign the stale fetched retryCount plus one. Overlapping cron invocations can overwrite one another rather than atomically incrementing the retry count.",
"evidence": [
"https://github.com/calcom/cal.com/blob/c790227e0cda780e6fea9bb03af27948d9e286b9/packages/features/ee/workflows/api/scheduleSMSReminders.ts#L181",
"https://github.com/calcom/cal.com/compare/917c7b0764f4c6bfdca95db2c5f73dbfdbe11757...c790227e0cda780e6fea9bb03af27948d9e286b9"
]
}
},
{
"file": "packages/features/ee/workflows/api/scheduleSMSReminders.ts",
Expand All @@ -21,7 +31,17 @@
"severity": "high",
"category": "robustness",
"crb_category": "error-handling",
"description": "The deletion logic in scheduleSMSReminders.ts incorrectly deletes non-SMS workflow reminders (e.g., Email, WhatsApp) that have retryCount > 1. This occurs because the retryCount condition within the OR clause for deletion lacks a method: WorkflowMethods.SMS filter, causing it to apply to all reminder types instead of only SMS reminders, which is the intended scope of this function."
"description": "The deletion logic in scheduleSMSReminders.ts incorrectly deletes non-SMS workflow reminders (e.g., Email, WhatsApp) that have retryCount > 1. This occurs because the retryCount condition within the OR clause for deletion lacks a method: WorkflowMethods.SMS filter, causing it to apply to all reminder types instead of only SMS reminders, which is the intended scope of this function.",
"adjudication": {
"date": "2026-10-02",
"status": "real-miss",
"reason": "filter-scope",
"why": "The new retryCount > 1 OR branch lacks method: SMS, so deleteMany applies it to reminders for every method. The previous method filter scoped the whole delete to SMS.",
"evidence": [
"https://github.com/calcom/cal.com/blob/c790227e0cda780e6fea9bb03af27948d9e286b9/packages/features/ee/workflows/api/scheduleSMSReminders.ts#L38",
"https://github.com/calcom/cal.com/compare/917c7b0764f4c6bfdca95db2c5f73dbfdbe11757...c790227e0cda780e6fea9bb03af27948d9e286b9"
]
}
}
]
}
24 changes: 22 additions & 2 deletions eval/corpus/labels/crb-discourse-graphite-9.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,17 @@
"severity": "low",
"category": "correctness",
"crb_category": "bug",
"description": "Thread-safety issue with lazy @loaded_locales"
"description": "Thread-safety issue with lazy @loaded_locales",
"adjudication": {
"date": "2026-10-02",
"status": "label-wrong",
"reason": "refuted",
"why": "load_locale already holds LOAD_MUTEX and rechecks membership inside the lock. The label supplies no unsynchronized write or reproducible new duplicate-load race for normally initialized locale state.",
"evidence": [
"https://github.com/ai-code-review-evaluation/discourse-graphite/blob/ecfa17b5a79dfdc91e7a4d50b42ae78a35d0a293/lib/freedom_patches/translate_accelerator.rb#L45",
"https://github.com/ai-code-review-evaluation/discourse-graphite/compare/728845d008f06828630a632e5c7d67e11bf15deb...ecfa17b5a79dfdc91e7a4d50b42ae78a35d0a293"
]
}
},
{
"file": null,
Expand All @@ -21,7 +31,17 @@
"severity": "low",
"category": "correctness",
"crb_category": "bug",
"description": "Consider normalizing the input locale (e.g., to a symbol) when checking/loading here to avoid double-loading if the same locale is passed as a String vs Symbol (also applies to other locations in the PR)."
"description": "Consider normalizing the input locale (e.g., to a symbol) when checking/loading here to avoid double-loading if the same locale is passed as a String vs Symbol (also applies to other locations in the PR).",
"adjudication": {
"date": "2026-10-02",
"status": "label-wrong",
"reason": "unreachable-input",
"why": "The new caller supplies I18n.locale and symbol-valued fallback locales; it does not introduce a caller passing a string. Existing locale loading already used the same array. No new mixed-string/symbol loading path is established.",
"evidence": [
"https://github.com/ai-code-review-evaluation/discourse-graphite/blob/ecfa17b5a79dfdc91e7a4d50b42ae78a35d0a293/config/initializers/i18n.rb#L6",
"https://github.com/ai-code-review-evaluation/discourse-graphite/compare/728845d008f06828630a632e5c7d67e11bf15deb...ecfa17b5a79dfdc91e7a4d50b42ae78a35d0a293"
]
}
}
]
}
Loading
Loading