[vincent-peng]

Bilibili BV video pages fall through to page text or inaccessible blob-media transcription. Read captions inside the active tab using its existing Bilibili session, resolve the selected p= part, and pass timestamped captions to summary and cached follow-up chat.

Relates to #533. Independent of CLI PR #541 and targets main directly.

• Prefer human English, AI English, Chinese, then other tracks. Retain language and generated-caption provenance.
• Keep authenticated metadata requests inside the page. Caption payload requests omit credentials, restrict hosts, reject redirects, and have bounded timeouts. Video/part changes invalidate results.
• Honor explicit Page mode. Missing captions do not become video-description summaries.
• Preserve the caption error when audio recovery also fails. Show localized Bilibili login/Page guidance directly; successful audio recovery clears the old error.

Inspectable behavior and upgrade proof

Actual request/output metrics, visible recovery guidance, before/after comparison, real-model summary/chat, and released-settings upgrade logs.

Live retrieval in logged-in Chrome returned five AI tracks on BV1ipnr6cExW; without credentials, the same metadata request returned none. English contains 267 cues through 562.8 seconds.

The production extension then ran in a disposable profile with the captured real caption payload replayed at the network boundary. Its Direct-provider client used a loopback compatibility bridge to the real configured cli/agy backend, producing a 994-character summary and 992-character follow-up. Chat retained all 267 timed cues exactly and bilibili-ai-captions/ai-en provenance. The official v0.25.3 extension failed the same caption-input contract. No Bilibili cookies or real provider keys were copied into the test profile.

An official v0.25.3 release ZIP, verified against its asset digest and SHA256SUMS, wrote 34 explicit settings in an isolated profile. Replacing it at the same path/profile/extension ID with the fixed production build retained every stored field and actual Options control. Fresh defaults were checked independently; an explicitly test-only version-update reload also retained settings.

Validation

• pnpm run build
• pnpm run check: 3,909 passed, 43 skipped
• Two failure-path regressions failed before repair; preparation/summary suites now pass 38 tests
• pnpm -C apps/chrome-extension test:chrome: 3 production checks, then 120 passed and 8 optional/live skips
• Restored production build, restarted/authenticated daemon, reran the Bilibili production-manifest E2E successfully
• Localization and typechecks passed for all 14 catalogs

Limits and maintainer decision

The live fixture has one part; the p=2 browser test uses controlled metadata to verify routing. JSON inventory from /x/player/wbi/v2 is supported. Protobuf-only inventories from /x/v2/subtitle/web/view remain unsupported. Daemon slide extraction retains its URL pipeline.

Live authenticated retrieval and production-extension summary/chat replay are separate proofs. A same-profile live authenticated extension run remains unverified because browser security policy blocked extension-manager access.

Maintainer approval of automatic caption routing and English-first ordering remains open. Recommendation: accept this bounded initial scope, retain explicit Page mode, and document the protobuf-only boundary.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: blocked before merge.

What this changes

The extension retrieves timestamped Bilibili captions through the active tab’s session for summaries and follow-up chat.

Example: Summarize BV1ipnr6cExW in Browser mode while signed in to Bilibili.

• Before: The extension uses page metadata or attempts transcription of inaccessible blob media.
• After: The extension supplies 267 timestamped English AI-caption cues to summary and cached chat context.

Review scores

Measure	Result	What it means
Overall readiness	🦐 gold shrimp (3/6)	The recovery finding is resolved and supplied behavior and upgrade evidence are sufficient; readiness remains capped by the unresolved product choices.
Proof confidence	🐚 platinum hermit (4/6)	Sufficient (live_output): The supplied output exercises the production extension’s caption-to-summary/chat path with captured real captions, a real model backend, unavailable-caption recovery, and explicit Page recovery; live authenticated retrieval is separately demonstrated, while same-profile live integration and live multipart captions are not claimed.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Product

Kind: Feature · Worth it: Needs a maintainer decision · Fix scope: Partial
User problem: Browser-only users cannot summarize the spoken content of captioned Bilibili videos when the media source is inaccessible outside the page.
Reason: The supplied fixture demonstrates value, but automatic routing and language preference are owner choices, and protobuf-only inventories remain outside scope.

Merge readiness

⛔ Blocked before merge - 1 item remains

This PR adds a useful capability absent from main and fixes the previous recovery finding; automatic routing, caption-language ordering, and the initial support boundary still need an owner decision.

Priority: P2
Reviewed head: aa268dc01d3817507ed520232a3e1215c0c08469
Owner decision: Required. See Decision needed.

Decision needed

• Question: Should Bilibili BV pages automatically use JSON captions with English-first selection while protobuf-only inventories remain unsupported?
• Recommendation: Accept the bounded initial integration: Approve automatic JSON-caption routing and English-first ordering, retaining explicit Page mode and the stated protobuf-only limitation.
• Why: This adds a site capability and changes default input selection; neither this PR nor the linked issue records maintainer approval of those choices.

Before merge

• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Findings

None.

Agent review details

How this fits together

The extension’s background content-preparation pipeline takes the active video URL and input mode, retrieves captions inside the page, and supplies text, timestamps, and provenance to summary and chat owners.

flowchart TD
  A[Active Bilibili tab] --> B[Media extraction plan]
  B --> C[Page session caption retrieval]
  C --> D[Prepared text and timestamps]
  D --> E[Summary provider]
  D --> F[Cached chat context]
  C --> G[Audio recovery or Page guidance]

Loading

Technical review

Best possible solution:

Adopt session-local caption extraction through the existing preparation owner if the maintainer accepts automatic routing, English-first ordering, and JSON-only initial support.

Do we have a high-confidence way to reproduce the issue?

Current-main source establishes the missing Bilibili caption route, and the supplied released-extension comparison demonstrates metadata reaching the provider instead of the captioned fixture.

Is this the best way to solve the issue?

A page-local retriever fits the existing caption-preparation boundary, avoids exporting cookies, and reuses summary and chat owners without adding a competing pipeline.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against ac39889e961b.

Provenance checked

• Media extraction planning changes intended behavior with a stated reason (ccea21d: Centralize input mode and transcript routing in one extension planner; the PR retains that owner and explains its Bilibili-specific default change.)
• Browser-only extraction and recovery keeps the original intent (fix: make Chrome Browser runtime daemonless #301: Keep Browser runtime local and prevent silent daemon URL fallback; the new caption route and localized recovery retain that boundary.)
• Cached summary and chat transcript metadata keeps the original intent (b20e0ee: Centralize cached extract values; the PR supplies Bilibili provenance through the existing cached-extract contract.)
• Localized user guidance keeps the original intent (feat(localization): render typed messages across CLI and extension #479: Use typed localized messages rather than matching English output; the recovery message follows that contract across all catalogs.)

Testing

Proof path: shipped entry point.

Security

None.

Evidence

What I checked:

• Verified introduction and current-main boundary: The pinned main-to-head diff adds the Bilibili retriever and integrates it into existing preparation; main’s extraction planner has only a dedicated YouTube caption route. (apps/chrome-extension/src/lib/media-extraction-plan.ts:12, aa268dc01d38)
• Previous finding resolved: Failed audio recovery now retains the caption error and localized login/Page guidance; successful recovery clears both, with distinct preparation and summary regressions. (apps/chrome-extension/src/entrypoints/background/panel-content-preparation.ts:355, aa268dc01d38)
• Production behavior and upgrade evidence: vincent-peng supplied production-extension request/output observations, exact cached-transcript hashes, visible recovery output, and v0.25.3-to-head settings logs at feat(extension): read Bilibili captions from the active tab #542 (comment); live authenticated retrieval and controlled browser replay are explicitly separate. (aa268dc01d38)
• Concrete user request and separate CLI work: laolxun supplied the captioned fixture in Support Bilibili subtitles (including AI-generated Chinese captions) in browser-only mode #533; feat: support explicit Bilibili CLI transcripts #541 explicitly implements a separate CLI download/transcription path.
• Session and network boundaries: Credentialed requests use fixed Bilibili API URLs inside the page; caption requests omit credentials, allow only named HTTPS hosts, reject redirects, and check video/part identity before requests and after response bodies. (apps/chrome-extension/src/lib/bilibili-page-transcript.ts:74, aa268dc01d38)
• Feature history and local history limit: History traces extraction planning, preparation, browser runtime, and caption refactors primarily to Peter Steinberger; old-name background history was also inspected. One historical stat lookup encountered an unavailable promisor object, while the relevant creation patches and current-main blame remained readable. (apps/chrome-extension/src/lib/media-extraction-plan.ts:20, ac39889e961b)

Likely related people:

• Peter Steinberger: Raw commit ccea21d adds apps/chrome-extension/src/lib/media-extraction-plan.ts:18 relative to its recorded parents. This identifies author metadata, not feature responsibility or a PR merger. (role: source-line author; confidence: high; commits: ccea21d48e59; files: apps/chrome-extension/src/lib/media-extraction-plan.ts)

Review metrics

Metric	Value	Why it matters
Change distribution	Production +342/-17; tests +621; docs +8	The test growth covers session handling, part identity, caption validation, routing, and recovery across existing owners.

Labels

Label changes:

• add proof: sufficient: Contributor real behavior proof is sufficient.
• add rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
• add status: ⏳ waiting on maintainer: Only explicit maintainer approvals or decisions remain; merge is still blocked.
• remove rating: 🧂 unranked krab: Current PR rating is rating: 🦐 gold shrimp, so this older rating label is no longer current.
• remove status: 📣 needs proof: Current PR status label is status: ⏳ waiting on maintainer.

Label justifications:

• P2: This is a bounded site integration with demonstrated user value and no established urgent regression.
• rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
• status: ⏳ waiting on maintainer: Only explicit maintainer approvals or decisions remain; merge is still blocked.
• proof: sufficient: Contributor real behavior proof is sufficient.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Record the maintainer’s decision on automatic routing, caption-language order, and the JSON-only support boundary.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (1 earlier review cycle)

• reviewed 2026-10-24T05:00:32.349Z sha 1a04c64 :: needs real behavior proof before merge. :: [P2] Preserve caption errors when audio fallback also fails

Reviewed October 24, 2026, 1:23 AM ET / 05:23 UTC (Revision 2).

[vincent-peng]

Addressed the caption-error finding in commit aa268dc. Preparation retains the caption failure and its localized descriptor through unsuccessful audio recovery. The panel displays login/Page guidance directly, without the generic Daemon recommendation. Successful audio recovery clears the old caption error.

Two regressions failed before the repair. The focused preparation/summary suites now pass 38 tests, localization/typechecks pass across 14 languages, and pnpm run check passes 3,909 tests with 43 skips. The supported Chrome gate passes 3 production checks plus 120 tests with 8 optional/live skips. After restoring the production build, the Bilibili E2E passed again and daemon restart/status authenticated successfully.

Production-extension behavior proof

Real-caption browser pipeline proof for #542

The failure-path repair is now in aa268dc01d3817507ed520232a3e1215c0c08469.

At 05:06:51 UTC on 2026-10-24, the existing logged-in Chrome session retrieved the real English AI captions for BV1ipnr6cExW: part 1, CID 42932375196, 267 cues. The same metadata request without credentials returned zero tracks.

I then exercised unchanged production extension builds in isolated Chromium profiles with that captured caption body. The page and metadata requests were controlled replays, not another live logged-in Bilibili session. No browser cookies or provider keys were copied. The ordinary Direct-provider client used a loopback test bridge that delegated to the actual configured cli/agy backend, not a stub summarizer.

Observed results at 05:14:08 UTC:

Official v0.25.3: zero Bilibili caption API calls; provider request contained page metadata instead of captured cues. Caption-input contract failed.
Fixed production build: no-tracks + blob-source fallback displayed Bilibili login/Page guidance, with zero model requests.
Explicit Page choice: recovered and generated a real model summary of the controlled page metadata.
Captured-caption extraction: real cli/agy summary, 994 characters; provider input contained first/last cues and timestamps.
Follow-up through visible chat controls: real cli/agy answer, 992 characters; full cached transcript exactly matched all 267 timed cues.

The source contract deliberately rejected the baseline's wrong-input request before invoking a model; it did not manufacture a successful baseline answer. Both production manifests were byte-identical, including permissions.

Observed model output excerpts, each a paraphrase of the video content:

Summary: The iPhone DUO rethinks foldable devices by uniting refined hardware with purpose-built software.
Follow-up: One major practical drawback mentioned in the review is the ergonomic strain caused by one-handed use.

The follow-up went on to cite 0:49–0:52, which matches the captured cues discussing pinky-finger strain.

{
  "captured_timed_transcript_sha256": "1605ebf0095cefb91f0d897935c5372eef45ac5376b02a21c9abeb01f4b943bb",
  "followup_cached_transcript_sha256": "1605ebf0095cefb91f0d897935c5372eef45ac5376b02a21c9abeb01f4b943bb",
  "exact_match": true,
  "cached_source_provenance": "bilibili-ai-captions/ai-en"
}

The visible failure message was:

Bilibili captions are unavailable in this browser session. Check that you are signed in to Bilibili, or choose Page to summarize the page text.

This establishes live session retrieval separately from production-extension summary/chat integration with the actual retrieved content. It does not claim a same-profile live authenticated extension run. This video has one real part; the p=2 routing test uses controlled metadata. Settings-upgrade proof is separate. The JSON-only boundary and English-first automatic routing remain explicit maintainer decisions.

Released-settings upgrade proof

Settings upgrade verification, 2026-10-24. The official v0.25.3 Chrome ZIP matched both its GitHub SHA256 digest and published SHA256SUMS. In disposable Chromium, its real service worker wrote 34 explicit synthetic settings. After closing Chromium, the extension directory was replaced by the current production build at the same path; the same browser profile and extension ID were retained. All stored fields and actual Options UI values remained unchanged.

Setting	Released profile after upgrade	Fresh head profile
UI locale	Turkish (tr)	Auto, resolved English
Output language	German (de)	Auto
Provider	Google	OpenAI
Model	google/gemini-2.5-flash	Auto
AI connection	Daemon	Direct
Media runtime	Daemon	Browser
Font size	18	14
Max characters	80,000	120,000

Both release and unmodified current build have manifest version 0.25.3. A separate artifact-only bump to 0.25.3.1 also loaded with the same ID and retained all settings; the canonical and immutable production manifests were not edited. No real keys, cookies, or user Chrome profile data were used. HTTP requests in the test profile were fulfilled locally.

{"phase":"seed-release","manifestVersion":"0.25.3","runtimeVersion":"0.25.3","extensionId":"ikcopffkdagfnphhcofccfaecmfomhkh","backgroundSha256":"231ce4de891b5c8acd39ffc44310bf379be63d7c1d5fbe4ea1066e4a7dab2b47","explicitSettingsRetained":34,"freshDefaultsVerified":false,"releasedStoredObjectRetainedExactly":null,"controls":{"htmlLang":"tr","uiLocale":"tr","languagePreset":"de","languageCustom":"","provider":"google","modelPreset":"custom","modelCustom":"google/gemini-2.5-flash","summaryRuntime":"daemon","slideRuntime":"daemon","fontFamily":"Georgia, serif","fontSize":"18","maxChars":"80000","promptOverride":"Synthetic upgrade fixture: retain this instruction.","colorMode":"dark"},"passed":true}
{"phase":"verify-head","manifestVersion":"0.25.3","runtimeVersion":"0.25.3","extensionId":"ikcopffkdagfnphhcofccfaecmfomhkh","backgroundSha256":"d454555fe9b5c46ff72124cc21f76433bd8f8ea0d135f2e68fcccbb64e12ead2","explicitSettingsRetained":34,"freshDefaultsVerified":false,"releasedStoredObjectRetainedExactly":true,"controls":{"htmlLang":"tr","uiLocale":"tr","languagePreset":"de","languageCustom":"","provider":"google","modelPreset":"custom","modelCustom":"google/gemini-2.5-flash","summaryRuntime":"daemon","slideRuntime":"daemon","fontFamily":"Georgia, serif","fontSize":"18","maxChars":"80000","promptOverride":"Synthetic upgrade fixture: retain this instruction.","colorMode":"dark"},"passed":true}
{"phase":"fresh-head","manifestVersion":"0.25.3","runtimeVersion":"0.25.3","extensionId":"ikcopffkdagfnphhcofccfaecmfomhkh","backgroundSha256":"d454555fe9b5c46ff72124cc21f76433bd8f8ea0d135f2e68fcccbb64e12ead2","explicitSettingsRetained":null,"freshDefaultsVerified":true,"releasedStoredObjectRetainedExactly":null,"controls":{"htmlLang":"en","uiLocale":"auto","languagePreset":"auto","languageCustom":"","provider":"openai","modelPreset":"auto","modelCustom":"","summaryRuntime":"direct","slideRuntime":"browser","fontFamily":"-apple-system, BlinkMacSystemFont, \"SF Pro Text\", system-ui, sans-serif","fontSize":"14","maxChars":"120000","promptOverride":"","colorMode":"system"},"passed":true}
{"phase":"verify-version-update","manifestVersion":"0.25.3.1","runtimeVersion":"0.25.3.1","extensionId":"ikcopffkdagfnphhcofccfaecmfomhkh","backgroundSha256":"d454555fe9b5c46ff72124cc21f76433bd8f8ea0d135f2e68fcccbb64e12ead2","explicitSettingsRetained":34,"freshDefaultsVerified":false,"releasedStoredObjectRetainedExactly":true,"controls":{"htmlLang":"tr","uiLocale":"tr","languagePreset":"de","languageCustom":"","provider":"google","modelPreset":"custom","modelCustom":"google/gemini-2.5-flash","summaryRuntime":"daemon","slideRuntime":"daemon","fontFamily":"Georgia, serif","fontSize":"18","maxChars":"80000","promptOverride":"Synthetic upgrade fixture: retain this instruction.","colorMode":"dark"},"passed":true}

Support boundary and maintainer decision

The real supplied video has one part. The part-2 browser test uses controlled metadata to exercise selected-part routing; it is not a claim of live part-2 captions on this video.

The tested production files were built from the repaired working tree subsequently committed as aa268dc. The immutable background SHA256 was d454555fe9b5c46ff72124cc21f76433bd8f8ea0d135f2e68fcccbb64e12ead2; the unmodified official release background SHA256 was 231ce4de891b5c8acd39ffc44310bf379be63d7c1d5fbe4ea1066e4a7dab2b47.

A same-profile live authenticated extension run remains unverified: browser security policy blocked the extension-manager page. We used the explicitly described split retrieval/replay proof and did not copy Bilibili cookies or change the user's installed extensions.

Maintainer approval of automatic JSON-caption routing and English-first ordering remains open. My recommendation is to accept this bounded initial support, retain explicit Page mode, and keep protobuf-only inventories documented as unsupported. CLI/downloader support remains separate in #541.

[vincent-peng]

@clawsweeper re-review

Head aa268dc01d3817507ed520232a3e1215c0c08469 preserves caption recovery guidance. The updated body links observable production-extension summary/chat and recovery output, release-to-head settings upgrade logs, and an explicit live-session/replay boundary. Maintainer approval of the bounded routing/language defaults remains open.

[clawsweeper]

🦞👀
Exact review queued.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/40066130840
• Updated: 2026-10-24T05:24:48.260Z