Rework BookWalker and Kodansha providers - #3
Merged
ThoughtzThruKeyz merged 1 commit intoAug 8, 2026
Conversation
Both providers are broken against the live services:
- api.kodansha.us no longer has DNS records at all.
- global.bookwalker.jp now 301s to bookwalker.com/migration/, which
serves a Cloudflare JS challenge, so server-side HTML scraping
cannot get through.
Kodansha now searches via the site's WordPress endpoint
(kodansha.us/wp-json/kodansha/v1/search-series) and parses series and
volume detail out of the server-rendered pages via KodanshaHtmlParser.
BookWalker now reads from BookWalker's own published catalog export
(static.bookwalker.com/data/bkwk-db.sqlite.zst) instead of scraping.
The export is verified as a zstd-compressed SQLite file, ~52 MB
compressed and ~233 MB extracted, regenerated daily. Downloads are
conditional on ETag/Last-Modified, land in generation-named files so
in-flight readers are not disturbed, and refresh in the background on a
24h interval. A cold start with nothing on disk blocks the first query
until the download completes.
Adds the zstd-jni dependency: commons-compress exposes
ZstdCompressorInputStream but treats the codec itself as optional.
Adds the bookwalkerDatabaseDir config option (default ./bookwalker).
Ported from https://github.com/loocool2/komf, source changes only. Their
build files, Dockerfile, release workflow, and README were deliberately
excluded: that fork is based three commits behind upstream and re-adds
the jvmToolchain(17) requirement that upstream removed in 8bcdea0. Their
MangaBaka changes were also excluded to keep this change scoped.
ThoughtzThruKeyz
marked this pull request as ready for review
August 8, 2026 00:21
5 tasks
ThoughtzThruKeyz
added a commit
that referenced
this pull request
Aug 8, 2026
Brings master up to date with work that landed in intermediate branches rather than here. PRs #3 and #4 were each merged into their stacked base branch after that base had already been merged to master, so neither reached master. Contents: - BookWalker provider reworked to read a locally downloaded catalog export (global.bookwalker.jp now redirects to a Cloudflare challenge, so scraping no longer works) - Kodansha provider reworked against kodansha.us (api.kodansha.us no longer resolves at all) - GitHub Actions workflow publishing a multi-arch image to GHCR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports the BookWalker and Kodansha provider reworks from loocool2/komf, source changes only.
Why
Both providers are broken against the live services — verified, not assumed:
api.kodansha.usglobal.bookwalker.jpbookwalker.com/migration/, which serves a Cloudflare JS challenge ("Just a moment…"). Server-side scraping cannot pass it.Neither was deprecated by upstream; both broke because the external services changed.
What changed
Kodansha — searches via the site's WordPress endpoint (
kodansha.us/wp-json/kodansha/v1/search-series, verified 200 JSON) and parses series/volume detail from server-rendered HTML via a newKodanshaHtmlParser. This trades a JSON API for scraping, which is inherently more fragile — but the API is gone, so it's scraping or nothing. Also fixes volume synopsis fetching.BookWalker — replaces scraping with BookWalker's own published catalog export at
static.bookwalker.com/data/bkwk-db.sqlite.zst. Volumes now render as"title, Vol. x".Verified against the live file:
application/zstd, 52,871,942 bytes,Last-Modifiedtoday, ETag present28 b5 2f fd; decompressed prefix isSQLite format 3contributors,labels,product_contributors,product_external_ids,products,publishers,series,series_tags,tags).series_ftsis the trigram index the downloader builds itself.Download behavior: conditional on ETag/Last-Modified, written to generation-named files so in-flight readers aren't disturbed, refreshed in the background on a 24h interval with failures non-fatal.
Things to know before running this
bookwalkerDatabaseDirconfig option, default./bookwalker.commons-iois used but declared nowhere — it resolves transitively throughcommons-compress. It packages fine today (443 classes in the shadowJar), but it's an undeclared dependency worth pinning later.Deliberately excluded
jvmToolchain(17), which upstream removed in8bcdea0and your fork already has.titlesv2 field is real (16 variants with language/is_primary) and would improve name matching. Their extensionless-cover fix is harmless but its premise didn't reproduce — the URL returns an identical 200 with and without.jpg..wasmblobs, the Google Drive extension link, the removed COEP/COOP headers, and a submodule pointer (bf519dcf) that exists in no public Komelia repo — sogit submodule update --initcannot succeed there.Test plan
:komf-core,:komf-mediaserver,:komf-appcompile clean:komf-app:shadowJarassembles; zstd-jni natives and commons-io both packaged