fix(repetition): skip detection for table-heavy output to prevent false positive loops#1574
Open
gorevyoneticisi wants to merge 1 commit into
Conversation
…se positives The sliding n-gram repetition detector triggers on markdown tables because table rows naturally have high n-gram overlap due to pipe separators and uniform row structure. This causes the agent to enter an infinite retry loop when generating structured tabular output with 5+ rows. Add isTableHeavy() check that detects when >40% of non-empty lines are table rows. When detected, skip consecutive repeat detection entirely. Adds 2 new tests: streaming large table (the exact scenario from XiaomiMiMo#1441) and table-then-new-content to verify no false positives. Closes XiaomiMiMo#1441
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The sliding n-gram repetition detector triggers false positives on markdown table output. Structured tables naturally have high n-gram overlap because of pipe separators (
|) and uniform row structure. This causes the agent to enter an infinite retry loop when generating tables with 5+ rows and 4+ columns (issue #1441).Root Cause
detectConsecutiveRepeat()scans for tokens that repeat at a fixed period. In a markdown table, each row starts with|followed by similar cell patterns, creating exactly the kind of periodic signal the detector looks for. The detector cannot distinguish "format-driven repetition" (table rows) from "content-driven repetition" (actual loops).Fix
Added
isTableHeavy()that detects when >40% of non-empty lines are table rows (lines starting with|or separator lines). When detected, skip consecutive repeat detection entirely. This is a clean, targeted fix:Changes
packages/opencode/src/session/prompt/text-ngram-detection.ts: AddedisTableHeavy()helper and gate inTextNgramMonitor.append()packages/opencode/test/session/text-ngram-detection.test.ts: Added 2 new tests covering streaming large tables and table-then-new-content scenariosTest Results
All 25 n-gram detection tests pass (23 existing + 2 new). No regressions.
Closes #1441