Skip to content

fix(channel): stop nil usage panics from crashing channel tests - #8039

Open
sylearn wants to merge 1 commit into
QuantumNous:mainfrom
sylearn:fix/channel-test-panic-isolation
Open

sylearn wants to merge 1 commit into
QuantumNous:mainfrom
sylearn:fix/channel-test-panic-isolation

Conversation

@sylearn

@sylearn sylearn commented Oct 26, 2026 •

Copy link
Copy Markdown

Agent

  • Tool: Claude Code
  • Tool version: 2.1.299
  • Model (full id): claude-opus-5-5
  • Host (CLI / IDE / GitHub coding agent / other): CLI
  • Date (UTC): 2026-10-26

本 PR 代码由 AI 辅助生成(提交者不是本仓库的历史核心开发者),已经人工确认方向。

Links

  • Closes #(暂无对应 issue;故障事实写在下方 Issue facts 中)
  • Related: none

User request

  • Verbatim:

我生产环境关闭了之后就好了,我觉得这个Bug,可以提交PR,根本上最佳修复好,你觉得呢,如果可以 最佳修复方式如何修呢

  • Later constraints or corrections from the user (quote, or none): 选择了“按上游约定提交,fork 后推送并创建 PR,描述附上脱敏后的崩溃栈”

Out of scope — refuse

  • Matched: no
  • If yes, what was told to the user (stop here; do not open a PR): n/a

Open gate — do not open unless all are satisfied

  • Out of scope (including pass-through-only forwarding): no
  • Usage / configuration / integration (answered instead of opening): no
  • Required issue facts present without invention: yes
  • Verification is actual commands or steps and observed results, not only go build or tests passed: yes
  • Body is short and factual; no unfiltered AI-generated text: yes
  • Open: yes
  • If no, what was told to the user (stop here): n/a

Kind

  • Bug fix
  • New feature
  • Performance / refactor
  • Docs
  • Other:

Issue facts

  • Actual behavior: 开启自动渠道检测(monitor_setting.auto_test_channel_enabled)后,后台检测 worker 发生 nil pointer panic,整个进程退出(Docker 退出码 2)。栈(已脱敏):
    panic: runtime error: invalid memory address or nil pointer dereference
    controller.testChannel                       controller/channel-test.go:499
    controller.testChannelForHealthCheck
    controller.performChannelTests.func1
    controller.runChannelTestWorkers.func1
    
  • Impact: 进程重启期间反向代理连不上 new-api:3000,API 与控制台同时 502。
  • Frequency: 生产环境 v1.0.0-rc.47(e835b1bf),检测每 3 分钟一次,并发 10;容器累计重启 42 次,留存的 42 份 panic 都在 499 行。
  • Evidence that the problem is in new-api rather than the client or upstream: panic 栈全部在 new-api 进程内;同一时刻代理日志出现 connect() failed (111: Connection refused);没有 OOM 记录。只关闭 auto_test_channel_enabled、不重启也不改代码后,15 分钟内没有新的 panic、重启或 502。
  • Applicable types and their fields (relay / billing / frontend / deployment; write "not applicable" otherwise): deployment — Docker,镜像 calciumion/new-api:v1.0.0-rc.47,PostgreSQL + Redis;relay / billing / frontend 不适用。

Change

  • coerceTestUsage:adaptor 的 DoResponse 可能返回装在 any 里的 (*dto.Usage)(nil),它不等于 nil 接口,会命中 case *dto.Usage 并被当作有效 usage 返回,随后在 499 行 usage.PromptTokens 处解引用。现在进入类型判断前先把它当作 nil 处理:流式请求按估算 usage 处理,非流式返回 usage is nil,与原有 nil 分支一致。
  • runChannelTestWorkers:worker goroutine 中的 panic 不会被 gin 的 recover 中间件捕获,会直接终止进程。现在每个渠道的检测都在 runChannelTestSafely 中 recover,记录渠道信息和栈,计为 1 次失败,不修改渠道状态,然后继续检测下一个渠道。

Research

Duplicate / prior art

Docs and code

  • https://docs.newapi.ai/ : 渠道文档只说明了手动“测试渠道 / 测试所有渠道”,没有涉及自动检测的异常处理。
  • https://deepwiki.com/QuantumNous/new-api : 访问时返回 HTTP 429,未能读取。
  • README / repo docs: 只说明了渠道测试的用法。
  • Code paths and what they imply for this change: controller/system_task_handlers.go 中,定时任务和手动“测试所有渠道”都通过 runChannelTestTask → performChannelTests → runChannelTestWorkers,因此两种触发方式都会崩溃,修在 worker 层可以同时覆盖。单渠道测试接口在 gin handler 内运行,有 recover 中间件,但同样会因为 typed nil 返回 500,第一处修复也覆盖了这种情况。

Alternatives considered

  • Option A: 只修 coerceTestUsage。能修掉这次的 panic,但以后其他 adaptor 再出类似问题,仍然会导致全站宕机。
  • Option B: 只在 worker 中 recover。可以止住崩溃,但 typed nil 仍会让测试结果出错。
  • Why this approach: 两处一起修:一处修正根因,另一处兜底,防止单个渠道的检测拖垮整个进程。

Files

Path Why
controller/channel-test.go typed nil usage 归一;worker 按渠道隔离 panic
controller/channel_test_panic_test.go 新增表驱动测试和 panic 隔离测试

Behavior

  • Before: 只要有一个渠道的测试返回 typed nil usage,整个 new-api 进程就会退出。
  • After: 该渠道计为测试失败(非流式)或使用估算 usage(流式);即使发生其他意外 panic,也只算作该渠道 1 次失败,并记录栈,进程保持运行,其余渠道继续检测。
  • Explicit non-goals / leftover work: 没有定位具体是哪个 adaptor 返回了 typed nil usage。

Verification

  • Commands and results:
    • 回退 channel-test.go 后运行 go test ./controller/ -run 'TestCoerceTestUsage|TestRunChannelTestWorkers' -count=1:--- FAIL: TestCoerceTestUsage,测试进程以 panic: runtime error: invalid memory address or nil pointer dereference 退出,与生产现象一致。
    • 加上修复后运行同一命令:ok github.com/QuantumNous/new-api/controller。
    • go vet ./controller/:无输出。gofmt -l controller/:无输出。
    • go test ./controller/ -count=1:ok github.com/QuantumNous/new-api/controller 35.233s。
  • Manual steps and observed result: 已在生产环境验证。基线为生产实际运行的 e835b1bf(rc.47),cherry-pick 本 PR 的提交(无冲突),按官方 Dockerfile 相同参数编译成镜像,然后重新开启 auto_test_channel_enabled(3 分钟,并发 10,passive_recovery):
    • 在该基线上运行 go test ./controller/ -count=1:ok ... 34.947s。
    • 前三个检测周期的 system_tasks.result:{"tested":14,"succeeded":7,"failed":7,"enabled":7}、{"tested":7,"succeeded":0,"failed":7}、{"tested":9,"succeeded":4,"failed":5,"enabled":4}。失败均为上游返回的 400/429。
    • 容器 RestartCount 为 0;日志中 ^panic 与 channel test panic 均为 0 次。修复前,同一配置下累计重启了 42 次。
  • UI: screenshot or recording (or why none): 不涉及 UI。
  • Tests added or updated, or why none: TestCoerceTestUsage 覆盖 nil、typed nil、指针、值、非法类型在流式和非流式下的组合;TestRunChannelTestWorkersContinuesAfterPanic 验证 panic 之后继续处理后续渠道,进度、汇总结果正确,且渠道状态不变。
  • Databases / providers / platforms exercised: 单元测试(macOS,本地 Go 环境);生产环境 Docker linux/amd64,PostgreSQL 15 + Redis 7,含 OpenAI 兼容类型的渠道。
  • Not verified: 生产验证期间,没有渠道再次返回 typed nil usage,所以这条路径和 recover 兜底只由单元测试覆盖;未在 MySQL / PostgreSQL / SQLite 上验证(本改动不涉及数据库)。

Risks

  • Failure modes: recover 会让逻辑缺陷不再导致进程退出,但会在日志中输出带栈的 channel test panic,需要通过日志发现这类问题。
  • Billing / quota / auth impact: 只有出现 typed nil 的非流式测试不再写入消费日志(原来会直接崩溃);其他计费逻辑没有变化。
  • Follow-ups: 排查是哪些 adaptor 会返回 typed nil usage。

Scope check

  • Single focused change: yes
  • Secrets included: no
  • Out of scope (Coding Plan / reverse-engineered channel / third-party wrapper / Codex / pass-through-only forwarding): no

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes
    • Channel tests now continue running when an individual test encounters an unexpected failure, and the affected result is counted as failed without changing the channel’s status.
    • Tests that receive a response without usage information now handle it consistently: non-streaming tests report a missing-usage error, while streaming tests use an estimated-token fallback.
  • Tests
    • Added coverage for missing usage information, supported usage formats, and recovery after a test failure.

coerceTestUsage returned a typed nil *dto.Usage as a valid usage, so
testChannel dereferenced it and the scheduled channel-test worker
goroutine panicked, killing the whole process.

Treat a typed nil usage like a nil usage, and recover panics per channel
in runChannelTestWorkers so one bad test is counted as failed without
changing the channel status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 26, 2026 15:06

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0b93a5a1-3e4c-4873-b00b-a8b8c33fd593
Commits

Reviewing files that changed from the base of the PR and between 34003e9 and f2f736f.

Files selected for processing: 2
  • controller/channel-test.go
  • controller/channel_test_panic_test.go

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

Channel tests now handle typed nil usage values through existing nil-usage behavior. Workers recover from panics, log channel details and a stack trace, and record a failed test without changing channel status.

Changes

Channel test safety

Layer / File(s) Summary
Usage conversion and validation
controller/channel-test.go, controller/channel_test_panic_test.go
coerceTestUsage normalizes typed nil *dto.Usage values. Tests cover XAI responses without usage and pointer, value, nil, and invalid usage inputs in streaming and non-streaming cases.
Worker panic recovery
controller/channel-test.go, controller/channel_test_panic_test.go
Workers call runChannelTestSafely. The helper recovers from panics and returns one tested, failed result. A worker test checks progress, summary counts, continued processing, and unchanged channel status.

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix

Possibly related PRs

Suggested reviewers: seefs001

Merge Risk: Minimal · up to f2f73

Recovered channel-test panics are reported as failed tests while other channels continue. No actionable merge-blocking risk was identified.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 28.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 2 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the main change: preventing nil usage panics from crashing channel tests.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the channels one by one
When usage hides, the tests still run
If panic leaps, the worker keeps its pace
A failed result takes its proper place
The stack trace leaves a trail to see
Then hops along, calm as can be

Comment @coderabbitai help to get the list of available commands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants