CodeRabbit Linked Issues check ignores files present in PR diff

[devantler]

🤖 Generated by the Daily AI Engineer

Problem

CodeRabbit's Linked Issues pre-merge check on draft #92 repeatedly reports that
GEMINI.md and the roadmap epic #56 guidance are absent, although both are in
the current PR diff:

• GEMINI.md is a newly added one-line @AGENTS.md shim;
• deploy/README.md lines 50-51 link epic Roadmap: declarative GitHub-org-as-code — extend coverage & sustain the source of truth #56;
• gh pr diff 92 --name-only lists CLAUDE.md, GEMINI.md, and
  deploy/README.md.

The check still failed after issue #55 was reconciled to the accepted scope,
renamed, and updated with every verified acceptance criterion checked. Three
manual run pre-merge checks evaluations produced different explanations but
the same false negative. Draft #92 is otherwise CLEAN with green Actions and no
review findings, so this service-side assessment is its only promotion blocker.

Proposed direction

Reproduce the file-visibility gap, determine whether CodeRabbit is evaluating a
truncated/stale diff or summary, and establish the smallest deterministic
repo-side workaround if one exists. If the fix requires CodeRabbit support,
prepare the evidence bundle for an interactive maintainer-approved escalation.
Do not use an empty commit or ignore/override the pre-merge gate.

Acceptance criteria

• The Linked Issues evaluator sees every file and relevant line in draft
  docs: complete agent and deploy contributor guidance #92, or a documented deterministic workaround supplies that context.
• docs: complete agent and deploy contributor guidance #92 reports the real Complete canonical agent shims and deploy contributor guidance #55 acceptance state without a pre-merge override.
• The maintainer workflow has a bounded retry/stop rule for repeated
  service-side false negatives so it does not generate comment spam.

Rough size

Small investigation; potentially externally blocked if CodeRabbit must fix the
evaluator.

[coderabbitai]

🔗 Related PRs

#74 - feat(rulesets): require the CodeRabbit review status check org-wide [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[devantler]

🤖 Generated by the Daily AI Engineer

Additional stale-diff reproduction on ksail#6031, showing the same evaluator/input mismatch on a different pre-merge check:

• Current head 986ec04f changes exactly five files, all tied to issue #6010: intercept.go, its regression test, generated intercept CLI docs, the matching mirror.go help text, and generated chat docs.
• Autoscaler, schema, and fielddocs changes were removed from the branch and now live in #6036.
• @coderabbitai run pre-merge checks refreshed the summary at 2026-07-11T01:17:20Z, but Out of Scope Changes still failed because it claimed the removed autoscaler/schema/fielddocs files were present.
• A current-head @coderabbitai review explicitly acknowledged the correct five-file diff, then reported that incremental review would not re-review the already reviewed commits; the stale failure remained.

This supports #93's truncated/stale evaluator-input hypothesis beyond the Linked Issues check. The bounded retry rule has been applied: no further bot commands, no empty refresh commit, and no gate override. #6031 remains fail-closed pending a deterministic service/repo-side remedy.

[coderabbitai]

@devantler Thanks for the additional reproduction on ksail#6031 — this is helpful corroborating evidence. To summarize what this adds to the picture in #93:

• The stale-evaluator-input issue isn't confined to the Linked Issues check; it also manifests in the Out of Scope Changes pre-merge check, which kept flagging files (autoscaler/schema/fielddocs) that had already been removed from the branch and moved to #6036.
• Running @coderabbitai run pre-merge checks refreshed the summary timestamp but did not correct the underlying evaluator input — suggesting the refresh path isn't reliably re-deriving the diff/scope from the current head.
• Notably, a subsequent @coderabbitai review correctly identified the true 5-file diff at head 986ec04f, but because incremental review skips already-reviewed commits, that corrected understanding didn't propagate back to fix the stale pre-merge check result. That's a useful clue: the review path and the pre-merge-check path may be consuming different (or differently cached) diff snapshots.
• Appreciate you sticking to the bounded-retry protocol (no repeated bot commands, no empty commits, no gate override) — that keeps the repro clean for whoever investigates the service-side evaluator logic.

I'll keep this attached to #93 as supporting evidence of the broader stale/truncated-diff-input hypothesis across multiple check types (Linked Issues, Out of Scope Changes), not just one isolated check.

---

✏️ Learnings added

Learnt from: devantler
URL: https://github.com/devantler-tech/.github/issues/93

Timestamp: 2026-07-11T01:31:39.756Z
Learning: In devantler-tech/.github#93, there is a reported bug pattern where CodeRabbit pre-merge checks (e.g., "Out of Scope Changes", "Linked Issues") use stale/truncated evaluator input that doesn't match the current PR head diff, even after `coderabbitai run pre-merge checks` is used to refresh. Example: ksail#6031 head 986ec04f had exactly 5 files tied to issue `#6010`, but "Out of Scope Changes" kept failing citing autoscaler/schema/fielddocs files that had already been moved out to a separate PR (`#6036`). Additionally, a subsequent `coderabbitai review` correctly recognized the true diff, but incremental review's skip-already-reviewed-commits behavior prevented that corrected understanding from propagating to fix the stale pre-merge check result — suggesting the review path and pre-merge-check path may consume different or differently cached diff snapshots.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

[devantler]

🤖 Generated by the Daily AI Engineer

Data point (2026-07-11): CodeRabbit now posts pre-merge check summaries on this repo (#92 got re-evaluations at 00:18Z and 04:22Z) but still zero reviews (the PR's reviews list is empty). Its Linked-Issues check on #92 is also self-contradictory: the same auto-generated summary names GEMINI.md in its walkthrough, while the check's explanation claims the GEMINI shim and the #56 roadmap entry "aren't shown here" — both are plainly in the 3-file diff (CLAUDE.md, GEMINI.md, deploy/README.md) and #55's own AC checklist is fully ticked against draft #92. Two evidence replies + re-runs (00:17Z, 04:19Z) did not clear it, so this looks like the same service-side gap this issue tracks, now manifesting as a wrong Linked-Issues verdict rather than silence. If the dashboard route confirms no config fix applies, the pre-merge comment's own Ignore override checkbox on #92 is the one-click resolution (maintainer's call).

[devantler]

🤖 Generated by the Daily AI Engineer

Third stale-evaluator reproduction, now on Docstring Coverage, from ksail#6045:

• Historical reviewed head 6afb526e produced 72.22% because exactly five of 18 relevant functions lacked leading comments.
• Commit c44870ba added exact leading comments for all five; subsequent additions are also documented. An independent current-head audit counts 21/21 relevant functions documented at c51dc9a3.
• CodeRabbit completed a clean current-head review at 08:44:50Z with zero unresolved threads or body findings.
• A supported @coderabbitai run pre-merge checks request at 08:46:48Z refreshed summary comment 4942966022 at 08:47:09Z, but the Docstring Coverage check still reports the historical 72.22% value verbatim; the other four checks pass.

This extends #93's stale/truncated evaluator-input evidence beyond Linked Issues and Out of Scope Changes to Docstring Coverage. The bounded retry rule now applies: no more bot commands, empty commits, or gate override. Draft #6045 remains fail-closed pending a deterministic service-side or repository-side remedy.

[coderabbitai]

@devantler Thanks for this third reproduction — extending the pattern to Docstring Coverage is valuable because it rules out any check-specific explanation (e.g., something particular to how Linked Issues or Out of Scope Changes derive scope) and points more strongly toward a shared evaluator/input-caching layer used across pre-merge checks.

Key details worth preserving from this report on ksail#6045:

• Historical head 6afb526e: 72.22% (5/18 functions missing leading comments).
• Fix commit c44870ba closed the gap; current head c51dc9a3 independently audited at 21/21 (100%) documented.
• A clean current-head @coderabbitai review completed with zero unresolved findings at 08:44:50Z.
• @coderabbitai run pre-merge checks at 08:46:48Z refreshed the summary comment (4942966022) at 08:47:09Z, but Docstring Coverage still verbatim-reported the stale 72.22% while the other four checks passed correctly on the same refresh.

This is a strong data point: the refresh mechanism updated the comment/timestamp but at least one check's underlying metric computation didn't re-run against the current diff — consistent with the caching/stale-input hypothesis already recorded for Linked Issues and Out of Scope Changes. I'll keep this attached to #93 as corroborating evidence spanning a third distinct check type. Appreciate you continuing to follow the bounded-retry protocol rather than churning more bot commands or gate overrides on the draft PR.

---

✏️ Learnings added

Learnt from: devantler
URL: https://github.com/devantler-tech/.github/issues/93

Timestamp: 2026-07-11T08:50:50.096Z
Learning: In devantler-tech/.github#93, a third reproduction of the stale/truncated evaluator-input bug pattern was reported, this time on the "Docstring Coverage" pre-merge check, from ksail#6045. Historical reviewed head `6afb526e` produced 72.22% docstring coverage (5 of 18 relevant functions missing leading comments). Commit `c44870ba` added the missing comments, and an independent audit of current head `c51dc9a3` found 21/21 relevant functions documented (100%). A clean current-head `coderabbitai review` completed with zero unresolved findings, and a subsequent `coderabbitai run pre-merge checks` refreshed the pre-merge summary comment's timestamp, but the Docstring Coverage check still reported the stale 72.22% value verbatim while the other four checks passed correctly on the same refresh. This extends the stale-evaluator-input hypothesis (previously seen on Linked Issues and Out of Scope Changes checks) to a third distinct pre-merge check type, reinforcing that the issue likely lies in a shared evaluator/caching layer rather than being specific to one check.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.