Repository navigation
feat: support explicit Bilibili CLI transcripts - #541
vincent-peng wants to merge 2 commits into
Conversation
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
clawsweeper
Bot
added
clawsweeper:intake:feature
|
Codex review: blocked before merge. What this changesThis PR adds explicit Bilibili BV video transcription through the existing downloader pipeline, preserving selected parts and validating cached recording identity. Example: Extract https://www.bilibili.com/en/video/BV1js411R7qA/?p=2 with --video-mode transcript.
Review scores
ProductKind: Feature · Worth it: Needs a maintainer decision · Fix scope: Complete Merge readiness⛔ Blocked before merge - 1 item remains Keep open: the previous cache finding is fixed and no new actionable defect was found; shipping still needs a maintainer decision about the patched downloader requirement. Priority: P2 Before merge
FindingsNone. Agent review detailsHow this fits togetherThe CLI URL flow sends explicit transcript requests into core extraction, which validates cached identity, downloads recording audio through yt-dlp, and returns transcribed speech for extraction or summarization. flowchart TD
A[CLI transcript request] --> B[Core URL strategy]
B --> C[Recording identity and cache]
C --> D[Bilibili downloader adapter]
D --> E[Shared yt-dlp pipeline]
E --> F[Configured transcriber]
F --> G[Transcript or summary]
Technical reviewBest possible solution: Land the bounded CLI integration after a supported downloader repair ships or a maintainer explicitly accepts the documented patched-downloader requirement. Do we have a high-confidence way to reproduce the issue? Current-main routing lacks this Bilibili acquisition path, corroborating the contributor's released-CLI empty-content and description-only observations; this review executed no target code. Is this the best way to solve the issue? Reusing transcript strategies and the shared downloader is a focused solution; the adapter owns only Bilibili normalization, download arguments, and recording metadata. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against ac39889e961b. Provenance checked
TestingProof path: shipped entry point. SecurityNone. EvidenceWhat I checked:
Likely related people:
Review metrics
LabelsLabel changes:
Label justifications:
Rating scale6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior. WorkflowClawSweeper edits this one comment on every review. Comment HistoryReview history (1 earlier review cycle)
Reviewed October 24, 2026, 1:07 AM ET / 05:07 UTC (Revision 2). |
|
Addressed the cache finding in commit 4f3557b. Bilibili identity now comes from the cached entry even when no usable initial resolution exists, and the same validity check guards fallback. Verified matching expired entries can still fall back; bypass diagnostics remain bypassed. Thirteen regressions failed before the fix. The targeted suites now pass 79 tests, the full check passes 3,922 tests with 43 skips, and structured Codex autoreview reports no accepted/actionable findings. Fresh built-CLI proofExecuted on 2026-10-24 from the fixed source tree. Every case uses both Runtime: Node 26.11.1, yt-dlp 2026.10.21 with yt-dlp/yt-dlp#17898 at pnpm run build
# YT_DLP_PATH selects the isolated patched executable for success cases.
# SUMMARIZE_WHISPER_CPP_BINARY and SUMMARIZE_WHISPER_CPP_MODEL_PATH select the local runtime.
# Both cookie environment settings are empty.
node dist/cli.js 'https://www.bilibili.com/video/BV1js411R7qA/' --video-mode transcript --json --format text --no-cache --no-media-cache --metrics off --timeout 2m --extract
node dist/cli.js 'https://www.bilibili.com/en/video/BV1js411R7qA/?p=2' --video-mode transcript --json --format text --no-cache --no-media-cache --metrics off --timeout 2m --extract
# Stock yt-dlp 2026.10.21, same extraction command as the first case: exits 1, stdout empty.
node dist/cli.js 'https://www.bilibili.com/video/BV1js411R7qA/' --video-mode transcript --json --format text --no-cache --no-media-cache --metrics off --timeout 2m --length short --language en --model cli/agy --force-summaryActual CLI output, filtered to transcript/summary results and redacted error details[
{
"case": "anonymous-extract",
"exit": 0,
"transcript_characters": 2158,
"transcript_sha256": "b948f15ac101ae247e8de7966a3089c67f3892e4ae9e29fc86cad2c4e74b0e57",
"transcript_source": "yt-dlp",
"transcription_provider": "whisper.cpp",
"summary_characters": 0,
"metadata": {
"provider": "generic",
"kind": "video",
"transcriptionProvider": "whisper.cpp",
"site": "bilibili",
"videoId": "BV1js411R7qA",
"part": 1,
"videoUrl": "https://www.bilibili.com/video/BV1js411R7qA/"
},
"transcript_excerpt": "Transcript: [MUSIC PLAYING] [APPLAUSE] I used to think of the"
},
{
"case": "language-part-extract",
"exit": 0,
"transcript_characters": 2156,
"transcript_sha256": "8cbf96004f6369370b77f2f13770bb3c1ac23ff9af45b61f31dd44d10ea05cf7",
"transcript_source": "yt-dlp",
"transcription_provider": "whisper.cpp",
"summary_characters": 0,
"metadata": {
"provider": "generic",
"kind": "video",
"transcriptionProvider": "whisper.cpp",
"site": "bilibili",
"videoId": "BV1js411R7qA",
"part": 2,
"videoUrl": "https://www.bilibili.com/video/BV1js411R7qA/?p=2"
}
},
{
"case": "stock-downloader-failure",
"exit": 1,
"transcript_characters": 0,
"transcript_sha256": null,
"transcript_source": null,
"transcription_provider": null,
"summary_characters": 0,
"metadata": null,
"stderr": "Failed to transcribe Bilibili video (Cache bypass requested; yt-dlp transcription failed: yt-dlp failed to download audio: yt-dlp exited with code 1: ERROR: [BiliBili] 1js411R7qA: Unable to extract initial state; please report this issue on <redacted-url> , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U)\n"
},
{
"case": "anonymous-summary",
"exit": 0,
"transcript_characters": 2158,
"transcript_sha256": "b948f15ac101ae247e8de7966a3089c67f3892e4ae9e29fc86cad2c4e74b0e57",
"transcript_source": "yt-dlp",
"transcription_provider": "whisper.cpp",
"summary_characters": 875,
"metadata": {
"provider": "generic",
"kind": "video",
"transcriptionProvider": "whisper.cpp",
"site": "bilibili",
"videoId": "BV1js411R7qA",
"part": 1,
"videoUrl": "https://www.bilibili.com/video/BV1js411R7qA/"
},
"summary_excerpt": "Most people unknowingly tie their shoes incorrectly because conventional instruction teaches an unbalanced knotting technique that leaves round laces prone to unraveling. After struggling to keep round nylon laces tied...",
"llm": {
"provider": "cli",
"model": "cli/agy",
"maxCompletionTokens": null,
"strategy": "single"
}
}
]Executable SHA256s:
The two video parts are subtitle variants of the same talk, so similar transcript lengths are expected. A fresh real downloader run observed distinct selected IDs and actual playurl CIDs: [
{
"requestedPart": 1,
"selectedId": "BV1js411R7qA_p1",
"selectedTitle": "【TED】如何系鞋带？ p01 简体中文",
"playurlCid": "5601020"
},
{
"requestedPart": 2,
"selectedId": "BV1js411R7qA_p2",
"selectedTitle": "【TED】如何系鞋带？ p02 英文",
"playurlCid": "5601811"
}
]Released-cache upgrade proofPersistent cache compatibility proof, 2026-10-24. Both installed 0.25.1 and latest npm stable 0.25.3 wrote isolated SQLite databases through their unmodified released cache APIs. Separate PR-source processes then read the same files. Cached content was synthetic; rejected Bilibili entries used real official yt-dlp 2026.10.21 failure, with no downloader mock and no browser cookies. The matching part-2 case uses current writeTranscriptCache, then closes and reopens the same database before resolution. Stored format version remains 2. No user cache/configuration was read or written. The released writer runs as plain Node importing its actual Maintainer decisionThe downloader shipping decision remains open. I rechecked the upstream PR and latest release: #17898 is still unmerged, and stock 2026.10.21 still fails the live fixture. My recommendation is to keep this PR unmerged until a supported downloader release contains the repair, unless a maintainer explicitly accepts the documented user-supplied patched-downloader setup. Browser caption support remains separate in #542. |
|
@clawsweeper re-review Updated head |
|
🦞👀 Re-review progress:
|
clawsweeper
Bot
added
clawsweeper:intake:feature
Bilibili recording pages can return empty content or a landing-page description for explicit transcript requests. This adds a bounded BV recording route through the existing yt-dlp and transcription pipeline.
summarize "https://www.bilibili.com/video/BV1js411R7qA/" --video-mode transcriptStandard and language-prefixed URLs retain the selected
ppart. Bilibili downloads use 1 MiB HTTP chunks. Cookies are forwarded only when explicitly configured through the existing cookie environment settings.Failed or empty acquisition raises an error before page-text fallback. Legacy description-only extracts and transcripts without a verified recording identity are refreshed. Identity validity is computed from the stored cache entry and enforced on both initial reuse and fallback, including expired and timestamp-ineligible entries. Matching expired transcript fallback remains available.
Browser caption retrieval is separate in #542. Guarded daemon downloader restrictions are unchanged.
Evidence and validation
Inspectable fixed-source CLI output, downloader revision/hashes, actual selected-part IDs/CIDs, and released-cache upgrade logs.
pnpm run buildandpnpm run checkpassed: 3,922 tests passed, 43 skipped.--no-cache --no-media-cachepassed for direct extraction, a language-prefixed URL selecting part 2, and summarization through local Whisper.cpp andcli/agy. Stock downloader failure exits 1 with empty stdout.Downloader compatibility and maintainer decision
Live success uses yt-dlp 2026.10.21 with yt-dlp/yt-dlp#17898 at
9fd15ea9ff813a78584db44457bae19023221f50. Stock 2026.10.21 still fails withUnable to extract initial state. As of 2026-10-24 the upstream repair is unmerged and no newer stable downloader release is available.Maintainer shipping approval remains open. Recommendation: keep this PR unmerged until a supported downloader release includes the repair, unless a maintainer explicitly accepts the documented user-supplied patched setup.
YT_DLP_PATHselects that executable; this PR adds no downloader dependency or installer.