Skip to content

Rework BookWalker and Kodansha providers - #3

Merged
ThoughtzThruKeyz merged 1 commit into
merge/upstream-prsfrom
feat/kodansha-bookwalker-rework
Aug 8, 2026
Merged

Rework BookWalker and Kodansha providers#3
ThoughtzThruKeyz merged 1 commit into
merge/upstream-prsfrom
feat/kodansha-bookwalker-rework

Conversation

@ThoughtzThruKeyz

@ThoughtzThruKeyz ThoughtzThruKeyz commented Aug 8, 2026

Copy link
Copy Markdown
Owner

Ports the BookWalker and Kodansha provider reworks from loocool2/komf, source changes only.

Based on merge/upstream-prs (PR #2), not master. Merge that one first, or retarget this to master and expect the earlier commits to come along.

Why

Both providers are broken against the live services — verified, not assumed:

Provider Old endpoint Status now
Kodansha api.kodansha.us No DNS records at all. Fully dead.
BookWalker global.bookwalker.jp 301 → bookwalker.com/migration/, which serves a Cloudflare JS challenge ("Just a moment…"). Server-side scraping cannot pass it.

Neither was deprecated by upstream; both broke because the external services changed.

What changed

Kodansha — searches via the site's WordPress endpoint (kodansha.us/wp-json/kodansha/v1/search-series, verified 200 JSON) and parses series/volume detail from server-rendered HTML via a new KodanshaHtmlParser. This trades a JSON API for scraping, which is inherently more fragile — but the API is gone, so it's scraping or nothing. Also fixes volume synopsis fetching.

BookWalker — replaces scraping with BookWalker's own published catalog export at static.bookwalker.com/data/bkwk-db.sqlite.zst. Volumes now render as "title, Vol. x".

Verified against the live file:

  • HTTP 200, application/zstd, 52,871,942 bytes, Last-Modified today, ETag present
  • zstd magic 28 b5 2f fd; decompressed prefix is SQLite format 3
  • Every table the new code queries exists in the real export (contributors, labels, product_contributors, product_external_ids, products, publishers, series, series_tags, tags). series_fts is the trigram index the downloader builds itself.

Download behavior: conditional on ETag/Last-Modified, written to generation-named files so in-flight readers aren't disturbed, refreshed in the background on a 24h interval with failures non-fatal.

Things to know before running this

  • Disk and bandwidth. ~52 MB download, ~233 MB extracted, plus ~14 MB index. New bookwalkerDatabaseDir config option, default ./bookwalker.
  • Cold start blocks. With nothing on disk, the first BookWalker query waits on the full download and decompress. Subsequent staleness refreshes are backgrounded.
  • commons-io is used but declared nowhere — it resolves transitively through commons-compress. It packages fine today (443 classes in the shadowJar), but it's an undeclared dependency worth pinning later.

Deliberately excluded

  • Their build files — that fork sits 3 commits behind upstream and re-adds jvmToolchain(17), which upstream removed in 8bcdea0 and your fork already has.
  • Their Dockerfile (Java 25, drops arm32), release workflow, and README.
  • Their MangaBaka changes, to keep this scoped. Worth revisiting: the titles v2 field is real (16 variants with language/is_primary) and would improve name matching. Their extensionless-cover fix is harmless but its premise didn't reproduce — the URL returns an identical 200 with and without .jpg.
  • Everything from PR Add Kavita 0.9.x+ compatibility fixes Snd-R/komf#309 that PR Merge upstream PRs #322, #316, and the code portion of #309 #2 already excluded. That fork merged Add Kavita 0.9.x+ compatibility fixes Snd-R/komf#309 wholesale and still carries the .wasm blobs, the Google Drive extension link, the removed COEP/COOP headers, and a submodule pointer (bf519dcf) that exists in no public Komelia repo — so git submodule update --init cannot succeed there.

Test plan

  • :komf-core, :komf-mediaserver, :komf-app compile clean
  • :komf-app:shadowJar assembles; zstd-jni natives and commons-io both packaged
  • Remote export verified as zstd SQLite with the expected schema
  • Confirmed no MangaBaka, Dockerfile, workflow, README, or submodule drift in the diff
  • Runtime: full download + first BookWalker match against a real library
  • Runtime: Kodansha search/match against a real library

Both providers are broken against the live services:

  - api.kodansha.us no longer has DNS records at all.
  - global.bookwalker.jp now 301s to bookwalker.com/migration/, which
    serves a Cloudflare JS challenge, so server-side HTML scraping
    cannot get through.

Kodansha now searches via the site's WordPress endpoint
(kodansha.us/wp-json/kodansha/v1/search-series) and parses series and
volume detail out of the server-rendered pages via KodanshaHtmlParser.

BookWalker now reads from BookWalker's own published catalog export
(static.bookwalker.com/data/bkwk-db.sqlite.zst) instead of scraping.
The export is verified as a zstd-compressed SQLite file, ~52 MB
compressed and ~233 MB extracted, regenerated daily. Downloads are
conditional on ETag/Last-Modified, land in generation-named files so
in-flight readers are not disturbed, and refresh in the background on a
24h interval. A cold start with nothing on disk blocks the first query
until the download completes.

Adds the zstd-jni dependency: commons-compress exposes
ZstdCompressorInputStream but treats the codec itself as optional.
Adds the bookwalkerDatabaseDir config option (default ./bookwalker).

Ported from https://github.com/loocool2/komf, source changes only. Their
build files, Dockerfile, release workflow, and README were deliberately
excluded: that fork is based three commits behind upstream and re-adds
the jvmToolchain(17) requirement that upstream removed in 8bcdea0. Their
MangaBaka changes were also excluded to keep this change scoped.
@ThoughtzThruKeyz
ThoughtzThruKeyz marked this pull request as ready for review August 8, 2026 00:21
@ThoughtzThruKeyz
ThoughtzThruKeyz merged commit bd71168 into merge/upstream-prs Aug 8, 2026
ThoughtzThruKeyz added a commit that referenced this pull request Aug 8, 2026
Brings master up to date with work that landed in intermediate branches
rather than here. PRs #3 and #4 were each merged into their stacked base
branch after that base had already been merged to master, so neither
reached master.

Contents:
  - BookWalker provider reworked to read a locally downloaded catalog
    export (global.bookwalker.jp now redirects to a Cloudflare challenge,
    so scraping no longer works)
  - Kodansha provider reworked against kodansha.us (api.kodansha.us no
    longer resolves at all)
  - GitHub Actions workflow publishing a multi-arch image to GHCR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant