Skip to content

Add zobs augment + fix free-form date year parsing - #6

Open
vivi365 wants to merge 6 commits into
mainfrom
augment
Open

vivi365 wants to merge 6 commits into
mainfrom
augment

Conversation

@vivi365

@vivi365 vivi365 commented Aug 31, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds zobs augment, a subcommand that fills in incomplete refs.bib entries
from bibliographic APIs, and fixes a year-parsing bug in build_bib_entry.

zobs with no subcommand still runs the sync unchanged.

zobs augment

Finds entries missing what an IEEE citation needs (venue, year, pages, arXiv
id…), resolves canonical metadata from Crossref → DataCite → DBLP → OpenAlex
→ arXiv
(identifier first, then a token_sort_ratio + first-author + year +
key-term match), fills refs.bib in place, and pushes the same fields back to
Zotero via update_item when the API key has write access.

  • Selection: interactive questionary checklist (all incomplete pre-checked),
    or --select KEYS, or --all. --json emits the candidate list for
    scripts / agents; --dry-run shows the diff and writes nothing.
  • Fills blanks only by default (--overwrite to force); repairs obviously
    malformed values (year = "June") unless --no-repair.
  • Entries with no confident match get a % zobs: no confident match — still missing {…} comment above them.
  • The key-term gate (MOVERY: / V1SCAN - must appear in a candidate title)
    rejects the false matches a bare fuzzy ratio would accept.
  • Responses cached under references/.zobs-cache/; requests spaced ~0.34s;
    a dead source never aborts a run.

New modules: cli.py, augment.py, metadata.py, cache.py.
New deps: rapidfuzz, questionary. Design: specs/augment-spec.md.

Year parsing fix

build_bib_entry took date[:4] as the year. Zotero dates are free-form, so
"June 2025" became {June}, "5/20/2017" became {5/20}. Replaced with
year_from_date(), which pulls the first 19xx/20xx from the string. Also
tightened build_bib_entry to emit volume/issue/pages/publisher/
place when present (needed so augment's pushed fields survive a re-sync) and
to format preprints as arXiv preprint arXiv:<id>.

Behaviour change

The next zobs sync after this lands will surface volume/pages/publisher
fields that were already in Zotero but not previously written, plus alignment
shifts on those entries.

Test plan

  • uv run pytest — 39 pass (new: test_augment.py, test_metadata.py)
  • uv run black --check . — clean
  • Dry-run against a live 58-item collection: 13 incomplete resolved via a mix
    of DOI and title search; false matches rejected by the key-term gate; blog
    posts correctly marked no-match. No writes to Zotero during testing.

https://claude.ai/code/session_01GmjZkdNscHJ5g7DfkSVZXn

Closes #7
Closes #9

build_bib_entry took the first 4 characters of the Zotero date field as
the year. Zotero stores dates free-form, so "June 2025" became {June},
"April 2026" became {Apri}, and "5/20/2017" became {5/20}. 15 entries in
one project bib were affected.

Replace the slice with year_from_date(), which pulls the first 19xx/20xx
match out of the string and returns "" when there is none.

Claude-Session: https://claude.ai/code/session_01GmjZkdNscHJ5g7DfkSVZXn
Authoritative description of the feature: completeness model, the
Crossref/DataCite/DBLP/OpenAlex/arXiv resolution pipeline, the matching
tiers, field mappings, refs.bib splice and Zotero write-back.

Claude-Session: https://claude.ai/code/session_01GmjZkdNscHJ5g7DfkSVZXn
New `zobs augment` subcommand (argparse front end in cli.py; bare `zobs`
still runs the sync). It finds entries missing what an IEEE citation needs,
resolves canonical metadata, writes it into refs.bib and — when the API
key allows — pushes the same fields back to Zotero.

- augment.py  — completeness model, interactive/​--select/​--json selection,
  plan_changes (fill blanks, repair malformed, respect --overwrite),
  refs.bib splice with no-match markers, Zotero update_item + 412 retry
- metadata.py — Metadata model, adapters for Crossref / DataCite / arXiv /
  DBLP / OpenAlex, identifier-first resolution then a token_sort_ratio +
  author + year + key-term match (the key-term gate rejects e.g. a VUDDY
  hit for a MOVERY query)
- cache.py     — keyed on-disk response cache, ~0.34s request spacing,
  swallows all network errors so one dead source never aborts a run
- sync.py      — build_bib_entry now emits volume/issue/pages/publisher/
  place when present and formats preprints as arXiv; shared generate_bib()

deps: rapidfuzz, questionary. Spec: specs/augment-spec.md.

Claude-Session: https://claude.ai/code/session_01GmjZkdNscHJ5g7DfkSVZXn
- default --sources is now dblp,crossref,datacite,openalex,arxiv so the
  title search asks DBLP before Crossref (better coverage for USENIX / NDSS
  / S&P; Crossref title search produced false hits on short titles)
- rewrite the README section in plain prose: drop the em dashes, the
  one-row variable table, and the "clears the bar" phrasing; spell out the
  DOI-vs-title-search order

Claude-Session: https://claude.ai/code/session_01GmjZkdNscHJ5g7DfkSVZXn
- README + .env.example: ZOBS_CONTACT_EMAIL is your own email address,
  optional, for the Crossref polite pool
- tighten the README augment section prose
- scrub em dashes from specs/augment-spec.md

Claude-Session: https://claude.ai/code/session_01GmjZkdNscHJ5g7DfkSVZXn
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add zobs discover: find papers your collection keeps citing but does not hold Group libraries unsupported: library type hardcoded to "user"

1 participant