Repository navigation
PageLedger 0.7.0: escalation ladder, contest and adjudication - #8
Merged
Merged
Conversation
A born-digital paper's math font came out as Ethiopic, Oriya, Tamil and Syriac letters, and 0.6 flagged two of its nine formula pages. foreign_script_characters fires on at least three letters or marks outside Latin, Cyrillic and Greek (letterlike and mathematical alphabets excepted) when they are under 10% of the page's letters. On the text layers of six test documents (252 pages) it fired on those nine pages and one with Hebrew points for a negation sign, and nowhere else. It holds the page as a coverage defect; the benchmark oracle recomputes the new count.
processing.escalate_on adds two triggers to 0.6's rule that a held page climbs. rough fires when the selected reading's share of known words falls below a threshold, judged by the pymorphy3 Russian lexicon (pageledger[ru]); OCR that read two columns as one scored 0.87-0.91 against 0.98-0.995 for clean text. disagreement climbs where clean readings disagree, which used to stop at review. Triggers lift a page at most to the image stage; a later reading that is not rough replaces a rough one. Each climb is recorded with its stage and evidence, the job records the lexicon's versions and verify-job refuses another, and the report adds work by stage. Configs without escalate_on compile to the same policy as before, so 0.6 jobs resume and verify unchanged.
With processing.contest, a page whose selected reading came from a model is compared with the best literal reading of the same page. Each difference is a span anchored in the model's text and kept in contested/<page_id>.json. Rule H settles look-alike letters and rule G settles RapidOCR's listed glyph confusions; a literal non-word those confusions do not explain stays open, since it may be the printer's misprint. Open spans hold the page as contested. On the two fully checked dogfood documents the rules settled 175 spans with no error, and the open spans covered 303 of the 319 corrections the model's text needed.
pageledger adjudicate --packets writes, for each page with open spans, both readings, the open spans and the exact image the model read. --decisions records the answers as adjudication receipts bound to the spans they answer. A high-confidence answer settles its span; when none is open the page is adjudicated_text, kept apart from reviewed_text, and its edition (the model's reading with the settled changes applied at their offsets) is what the transcript, document.json and exports carry. verify-job rebuilds each edition from its receipts. On a contested page the coverage, low-confidence and blank holds of readings the page no longer rests on are dropped; holds on the two compared readings stay.
… one word Checked on Большаков with Sol as the reader: in column order RapidOCR scored 0.95 to 0.97 known words while still misreading about one word in a hundred, so the rough trigger let six of seven pages stop at OCR. always sends every page to the reader, for jobs that contest its reading. Column order also carries the second half of a word hyphenated at a line end elsewhere; contest no longer reports that as two differences. Other splits and joins stay differences: a reader that writes «гос средств» for the printed «госсредств» has changed the text.
With every page sent to the reader, selection still kept clean OCR on six of Большаков's seven pages, so their readings were never compared. A job that contests readings now selects a clean reader reading when one exists; its differences from the OCR are the spans to settle.
A span now shows both readings in sorted order and the words around it, never which engine read which. On Большаков, Sol adjudicating spans labelled as its own reading kept three of its own substitutions and called each one plainly printed; Gemini 3.1 Pro, blind, left one error in 60 spans.
From an independent review of the 0.7 branch: - A single word read in another place is a difference: a model may move a negation. Only runs of two or more words, or a word alone on its line, count as reading order. Similar words pair within one replaced phrase, in order, or across blocks only where the witness read them at the same place. - Spans carry each engine's exact characters. Words only the witness has are anchored after the word it read before them; inserted words get a space and a removed word takes one with it. Answering every span with the witness rebuilds the witness's words. - The halves of a hyphenated word must sit together or keep the hyphen. - The ru-print-0.1 output is pinned by a golden test, and a job refuses, rather than rewrites, different spans from the same two readings. - A person accepting an adjudicated page accepts its edition; choosing another reading publishes that reading. - Answers already recorded are refused, and packets need an empty directory. - verify-job checks the contested, edition and trigger summaries. - Once every span is settled, the witness's own coverage and confidence holds are set aside; every hold set aside is recorded. - contest with a lexicon and no triggers no longer crashes. - The ru extra pins the lexicon versions jobs record, and CI runs it. - Packets and decisions have schemas; micro and ordinal signs are not foreign script.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PageLedger 0.7.0: the escalation ladder, in the reduced scope agreed for this release (climb, contest, settle on the existing four stages). 0.6.1 (#7) was merged and never published; its changes ship here and are folded into the 0.7.0 changelog.
What changes
processing.escalate_onaddsrough(Russian lexicon via the pinnedpageledger[ru]extra),disagreementandalwaysto 0.6's hold rule. Each trigger is recorded with its evidence; the report shows work by stage. Newforeign_script_characterswarning.processing.contest: {rules: ru-print-0.1}compares a model's reading with the literal OCR reading word by word. Every difference is a span incontested/<page_id>.json, in the model's offsets. Rule H (look-alike letters) and rule G (RapidOCR's listed glyph confusions) settle what the two readings decide. A literal non-word those confusions do not explain stays open, because it may be the printer's misprint. Open spans hold the page ascontested.pageledger adjudicate --packetswrites blind packets (both readings sorted, the image the model read).--decisionsrecords the answers as receipts bound to the span artifact's hash. When no span is open the page isadjudicated_text, kept apart fromreviewed_text, with an edition (the model's text with the settled changes applied) that the transcript and exports carry.verify-jobrebuilds spans and editions.Evidence
Verification
ruff,mypyclean.dev,pdf; 1536 passed withdev,pdf,ru.check_release.py v0.7.0agrees.twine checkandcheck_distributions.pypass. Both maintained tutorials pass against each, installed outside the checkout.dev,pdf,rulane, so the lexicon tests run in CI.