Repository navigation
Release 0.6.0: new engines, cross-engine checks, review sheets, and export - #6
Merged
Merged
Conversation
Work from the Kinodrive real-document survey and the documentation audit: - Skip hidden files (.DS_Store, ._*) in folder inputs; report skipped_inputs. - Normalize encoding names in replay profiles (UTF-8 vs utf-8 broke replay wherever LANG was unset). - Refuse runs with no extraction route (no taxonomy.page_types). - Reject run settings that process would ignore; warn on leftover adapter and taxonomy sections. - Typed PageLedger diagnostics and a pdf_ocr preflight for missing binaries and language packs, shown in full instead of redacted. - Bound pdf_ocr renders by max_render_pixels (render_dpi_capped, render_limit). - Read AES-encrypted PDFs that open without a password (pypdf[crypto]); refuse password-protected, non-standard-handler and damaged PDFs with unsupported_encryption, malformed_pdf or missing_crypto_dependency. - Warn on hollow text layers: digits_only_text, mixed_script_tokens, private_use_characters and run-level repeated_page_text. - Give low_confidence its own document-job hold, keeping older jobs' mapping through hold_policy. - Let page types set review: true to extract and still hold for review; init-config uses it for sparse, table_likely and unknown. - Pause document jobs on budget limits (paused_budget) and continue with resume --raise-limit, recorded in limits_history. - Write artifacts atomically; report hung OCR engines as engine_timeout. - Test documented configs, links, anchors, version strings and the README first run.
Some scanned PDFs declare pages far smaller than the paper, so a fixed 300 DPI rendered their images at half their own resolution. pdf_ocr now reads the pixel size of each page's embedded images (without decoding them) and renders at the largest image's resolution when that is higher, up to 1200 DPI. The model string records the reason. Each PDF is read once per adapter through a shared helper that refuses a source changed during the run.
corpus/manifest.public.yml lists 23 public-domain scans with their source, SHA-256 and the pages each tier runs; library-held items go in a git-ignored local manifest of the same shape. scripts/corpus/run.py checks each source against its hash and runs one verified document job per item, recording the engine, timing and package hash of every page attempt. compare.py lists the pages that changed between two runs, and sweep.py checks that every PDF under a folder opens, one child process per file.
pageledger export writes a verified job's selected text page by page with each page's disposition, review, attempt, engine and output hash, and can keep only reviewed pages. The TEI export is a minimal TEI P5 document with a page break per source page, reviewed and unreviewed pages marked apart, and a gap for pages without text. Links point to the source relative to the exported file. Document reports now name the source by its path relative to the job directory (report_format 0.6), so shared reports carry no absolute paths. Reports written by 0.5.1 keep their paths and still verify.
repetition_loop marks output from a model stuck in a loop: one line on much of the page, an ending repeated 20 or more times, or one letter repeated 40 times. On 287 outputs of 24 transcribed pages by 11 engines it flags 16 pages, every one a loop, and nothing from classic OCR, hosted models or the reference. A top-level language block enables script_mismatch and, for pre-reform text, historical_letters_lost. Quality lines record the loop evidence, the benchmark oracle recomputes it independently, and document jobs hold such pages as coverage defects.
pageledger review-sheet writes a CSV with one row per page (a link to the page, its disposition, the start of its text and its review reasons) and a binding file recording the evidence each row was written from. review-job imports the filled-in sheet: it checks every row against the job first, so a stale, duplicated or formula-bearing row stops the import before anything is recorded, and --dry-run checks and counts decisions without recording them. Decisions from a sheet need a named reviewer.
The vision adapter sends each page, rendered by Poppler as a JPEG within the request's size limit, to a vision model behind an OpenAI-compatible chat-completions endpoint, on this machine or hosted. Remote hosts need https and allow_remote; the key comes only from a named environment variable; redirects and environment proxies are refused. Finish states map to typed failures, adding MODEL_CONTENT_FILTERED and MODEL_RECITATION. In image stages it keeps the exact JPEG sent as image evidence, and every attempt is marked generative.
A regex in export.py and two quality tests held literal noncharacters and private-use code points, which are invisible in editors and diffs.
Document jobs compare every pair of completed readings of a page word by word, recording agreement, differing passages and differing numbers. Only comparisons between the selected reading and another clean one count, since a held attempt is why the page moved on. Agreement below 0.60 holds the page as engine_disagreement, a differing number as numeric_disagreement, and a generative reading no other engine confirms as unconfirmed_model_output. processing.benchmark runs a stage on every Nth page to compare engines on a sample. Jobs written before 0.6 are not compared and still verify.
scripts/corpus/score.py scores a folder of page texts, or a corpus run, line by line against reference transcriptions: code-point and grapheme CER, diplomatic and normalized, exact and missing lines, numbers per table cell, and pre-reform letters kept or added, overall, per stratum and for the worst pages, with loops, time and cost from runs. It follows the calibration grader: on all 11 calibration engines it reproduces the grader's CER within 0.005. Long engine lines are searched in overlapping pieces and HTML tables are read as rows, so no text is cut out of a match.
The corpus runner records each attempt's output file and failure code, and reads a child run's provenance once instead of once per attempt. The scorer's --stage option scores a stage's latest attempt on each page rather than the text the job selected, and counts that stage's failures by code, so a model run through process can be measured even where the job chose another text.
…cations A new warnings reference lists every quality warning, review hold and page disposition with its meaning, likely cause and next step. The specifications lead with what a user can rely on and move Python API detail into sections for adapter authors, and outdated or unimplemented claims from the 0.5.2 documentation audit are corrected.
…nning Three how-to guides walk through a real collection from sampling to export, choose OCR settings by language and script (including pre-reform print and image folders), and plan time and cost from measured engine speeds.
Troubleshooting is organized by symptom, from setup diagnostics to damaged PDFs and paused jobs. A citation guide says how to cite runs, jobs, transcripts and exports, what to deposit, and which files record absolute paths. The adapter tutorial now runs as written, and example configs load.
Reports marked 0.6 list pages needing a person first and show the selected text, engine agreement and human review as separate columns. Reports marked 0.5.1 and unmarked reports render byte for byte as before.
A glossary defines the terms the documentation relies on, including those 0.6 adds (engine agreement, generative, review sheet, selected text). The CLI reference gives every command's synopsis and exit codes, checked against the code, and documents how directory inputs are numbered and filtered. The capabilities, artifacts and design pages use plainer words, and AGENTS.md lists every file that pins the release version.
docs/engine-recipes.md gives install steps, a working config, measured results and licence for each engine a scholar can put behind PageLedger: local vision models through mlx-vlm or llama.cpp, RapidOCR with the pinned PP-OCRv5 Cyrillic recognizer, Tesseract orus, Apple Vision and hosted models. The RapidOCR example records its version and recognizer hash, reuses one engine per run and bounds rendering; both examples are tested end to end against fake engines.
…lane The README states what 0.6 does (engines, review in a spreadsheet, export, warnings), keeps its tested quickstart, says plainly what works on Windows, and links absolutely so it reads correctly on PyPI; a docs test checks those links against the repository. The docs index is grouped into tutorials, how-to guides, reference and explanation. The PDF/OCR tutorial reads a document.pdf, and CI runs it against a generated two-page scan through the tutorial runner's new --fixture option. Maintainer records moved to docs/maintainers/, and the text tutorial's behaviour checks moved from the page into its test.
The first-run tutorial now shows the lines a reader should see after the run, inspection and comparison. The PDF tutorial says what a finished OCR run leaves behind, why compare-runs does not total warning changes across adapters, and what each manifest status means. Specs and examples no longer name a package version where the schema version is the contract, and the version test also checks the manifest spec's JSON example.
On the two-page sample nothing is flagged, so the old rerun and compare-runs lines failed. The quickstart now classifies, runs with the route map, inspects and verifies, and points to the README's tested first run for rerun and compare-runs. The examples row lists the adapters that ship today.
The long list of built-in features is now grouped by what a reader wants to do, with tables where they read better. Detail that belongs to one feature now lives on that feature's page, and each group links there. The replay boundary keeps every limit, in plainer words; it now explains what an exact replay with no output shows and what evidence_compared means. Runs recorded before raw output was hashed are named as a limit: they cannot pass verify-run or be bundled.
The checkpoint specification now owns how an interrupted run is resumed, and the processing specification owns document-job recovery and limits. The README, CLI reference, artifacts guide, design notes, adapter protocol, rerun specification and agent skill keep what their readers need and link to the owner instead of restating the rules. The one rule only the CLI reference stated, that poor output is a job for rerun rather than resume, moved to the checkpoint specification. Words such as authoritative, canonical, generation-zero, dogfooded and non-hermetic give way to plain descriptions where a plain word says the same thing. The JFK example no longer calls its transcript a 0.4.1 transcript, since the version was not recorded.
…vision The README, the processing specification and docs/examples/processing.yml showed three different processing configs. They now match, with 100 attempted pages before a job pauses, and a test keeps them identical. The example's optional image stage and the processing specification now use the built-in vision adapter; the older OpenAI-compatible example stays for existing configs. A new test runs a document job whose empty page goes to a vision model in the image stage, and checks that the job keeps the image it sent and holds the model's reading for review.
A dry run does not load the adapter, so it counted real PDF pages only for pdf_text and pdf_ocr and planned one page for everything else. A dry run of the built-in vision adapter therefore planned one page per PDF, and --pages beyond the first was refused. It now counts pages as the vision adapter itself does. Found while planning a run on a 71-page census volume. The adapter protocol now says plainly how pages are counted without a page_count hook, and that a dry run cannot call a custom adapter's hook.
A hosted Gemini Flash model that reasons by default spent its whole output budget reasoning on dense pages: 9 of the 14 evaluation pages it read failed as MODEL_OUTPUT_TRUNCATED, and raising max_tokens to 32768 did not save the densest. The optional reasoning_effort (none, minimal, low, medium or high) is sent as the standard request field when set, and never otherwise. The adapter guide explains the failure and the two remedies.
Some gateways pass Gemini's finish reasons through unchanged, in upper case: MAX_TOKENS for a reading cut off at the token cap, SAFETY and its kin for a filtered page. The adapter compared them case-sensitively, so a cut-off page was reported as MODEL_INVALID_RESPONSE. It now compares in lower case and treats Gemini's block reasons as MODEL_CONTENT_FILTERED.
A vision page that failed in a plain run showed only 'AdapterFailure: <redacted>', and run.log said the same, so a reader plugging in a model could not tell a wrong address from a missing key or a reading cut off at the token cap. Typed failures carry a code and HTTP status that are checked against a fixed list, so the run now shows them, for example 'AdapterFailure: MODEL_QUOTA (HTTP 429)', whichever adapter raised the failure. Free-text adapter messages stay redacted. Resumable runs and document jobs name the code in their stop message as well.
The vision adapter guide now starts with the steps a reader takes: start or choose a server, write the config, read one page and look at it, then read the document. A table gives the address and model name of common servers (mlx_vlm, llama.cpp, Ollama, LM Studio, vLLM, gateways), marked by whether PageLedger was tested with them. New sections show how to change what the model is asked, what reasoning models need, and what a local model costs in time and heat, and how its loops end, from the G4 runs. The guide no longer claims that PageLedger holds every vision reading for review: document jobs do, a plain run does not unless the page type asks for it. The engine recipes carry the like-for-like G4 results, a hosted recipe with reasoning_effort, and no longer imply that Chandra was measured. The OCR options page drops its stale 'Cloud VLM needs a custom adapter' row and points its workflow at the vision adapter.
The troubleshooting guide gains a table of the MODEL_ failure codes, their likely causes and what to do: a server that is not listening, a base_url without its /v1, an unknown model name, a missing key, a quota, a timeout, a reading cut off by reasoning or a loop. Its opening no longer says that every adapter failure reads <redacted>, since typed failures now show their code.
The TEI validation test downloaded tei_all.rng from tei-c.org at test time and skipped whenever the site was unreachable, which from this machine was always. The schema of TEI P5 4.12.0 now lives in tests/fixtures/tei, copied unchanged from the TEI Consortium's release, with its source, checksums and licence in a README; a test pins its SHA-256. The export validates against it with xmllint, which the Python 3.13 CI lane now installs. The export guide says so.
Document jobs held a reading from a generative adapter such as vision as unconfirmed_model_output until another engine agreed, but a plain run let a clean model reading grade well and stay out of the review queue. A run now adds every page a generative adapter read to its review queue with that reason, taken from the extractor's recorded capabilities, and align re-derives it rather than duplicating it. Like other review reasons it makes the page rerun work, so rerunning with another engine and comparing the runs is the plain-run way to a second reading. Runs from adapters that are not generative are unchanged.
RapidOCR with a PP-OCRv5 recognizer was the best permissively licensed CPU engine measured, but it was an example that pip users had to fetch from GitHub and load with --adapter-path. It is now the built-in rapidocr adapter: pip install 'pageledger[rapidocr]' installs RapidOCR and ONNX Runtime, and a config names adapter: rapidocr with the recognizer directory. The adapter no longer hard-codes the Cyrillic language setting, which only mattered for downloads and right-to-left reordering, so any PP-OCRv5 recognizer works; provenance records its hash. The old recipe's 'rapidocr[onnxruntime]' installed no ONNX Runtime, since RapidOCR 3 declares no extras; the new extra lists both packages. doctor reports whether they are installed. Tested on a real 1888 page with RapidOCR 3.9.2, ONNX Runtime 1.30.0 and the Cyrillic recognizer.
The built-in vision adapter now reads pages in a document job's image and second-opinion stages and keeps the same image evidence, so the example OpenAI-compatible image adapter was a second, narrower way to do the same thing, and its defaults pointed at a maintainer's local gateway. It and its tests are removed. The image evidence specification keeps what any image adapter needs to know; the processing specification and adapter protocol no longer mention the example; the changelog says how to move a config to vision. The 0.5.1 validation record keeps its historical config.
Set the version to 0.6.0 in pyproject.toml, pageledger/_version.py, CITATION.cff and the editable entry in uv.lock; date the changelog and the citation 2026-09-25; move the pinned version assertions, the CI and publish smoke commands and the route-map and run-manifest examples to 0.6.0. The release checker agrees for v0.6.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PageLedger 0.6.0 lets you put stronger engines than Tesseract behind PageLedger, shows where they fail, and adds ways to review and share a job's text.
visionadapter reads pages with a vision model behind an OpenAI-compatible endpoint, on this machine or hosted. It sends pages to another machine only overhttps://withallow_remote: true, reports each failure with a typed code, keeps image evidence in document jobs, and passesreasoning_effortto models that reason before answering. It replaces the example image adapter, which is removed.rapidocradapter (pip install 'pageledger[rapidocr]') reads scans on the CPU with a PP-OCRv5 recognizer you supply, such as the Cyrillic one that keeps pre-reform letters.digits_only_text,mixed_script_tokens,private_use_characters,repeated_page_text) and the ways vision models fail (repetition_loop,script_mismatch,historical_letters_lost).unconfirmed_model_outputuntil another engine agrees or a person confirms it; a plainrunholds such pages too.review-sheetwrites a CSV for reviewing a job in a spreadsheet, andreview-job --reviewrecords its decisions after checking every row against its evidence.exportwrites a verified job's text as TXT, Markdown, JSONL or TEI P5.On 19 pages of Russian Empire and Soviet print, a hosted Gemini 3.8 Flash with reasoning turned off read all 19, differing from provisional model-made reference transcriptions in 3.8% of characters. A local Qwen3.5-9B run was stopped after 12 pages because the machine ran hot: 8 pages differed in 1.6% of characters, and 4 looped until they reached the token limit. On the 14-page failure-mode tier, 0.6.0 and 0.5.2 selected identical text on every page; the 174-page comparison with 0.5.2 was not run.
Validation: 1,434 tests passed, with the 3 stress tests deselected as the release procedure specifies; without the
pdfextra, 1,373 pass and 57 skip. Ruff lint and format, mypy and the release metadata check pass. The wheel and sdist, built once from 0146fc7, pass Twine and the inventory checks, and both maintained tutorials pass against each package installed in a fresh environment outside the checkout. TEI export validates against the pinned TEI All 4.12.0 schema. New artifact fields are optional, and all eight retained 0.5.1 document jobs passverify-jobunder the 0.6.0 wheel with their files unchanged. Core still depends only on PyYAML.