Warn if Linux automatic NUMA balancing is enabled at buffer init

[toffee-desuwa]

Summary

Adds a startup-time warning when /proc/sys/kernel/numa_balancing != 0, plus a new ## Performance gotchas section in the README documenting the mitigation. Fixes #624.

Background

@wenjianhn reported in #624 that Linux automatic NUMA balancing (kernel.numa_balancing != 0, default on most 5.x kernels) periodically rewrites PTEs at the user/kernel boundary via task_numa_work, which is directly visible on the deep_ep::Buffer::internode_dispatch hot path.

funclatency -d 60 -m -p <pid> task_numa_work over 60 s on an SGLang workload:

avg = 5 msecs, total: 789 msecs, count: 139
msecs distribution: 0-1: 72, 2-3: 8, 8-15: 57, 16-31: 2

139 invocations in 60 s, with 2 samples reaching the 16-31 ms bucket — added directly to the EP critical path.

Change

• deep_ep/__init__.py: new check_numa_balancing() helper, called between check_nccl_so() and init_jit() in the existing init block. Mirrors the check_nccl_so pattern (env-var suppress, silent skip on non-Linux, single warning emit). kernel.numa_balancing is a bitmask, so the check is value != '0' (covers modes 1, 2, and OR-ed combinations, all of which run task_numa_work).
• README.md: new ## Performance gotchas section documenting the issue and the sysctl kernel.numa_balancing=0 mitigation, plus an EP_SUPPRESS_NUMA_CHECK entry in the env var listing.

The check is opt-out via EP_SUPPRESS_NUMA_CHECK=1 for CI / containerized environments where the user cannot toggle the sysctl.

Validation

• Warning fires when kernel.numa_balancing is non-zero:

  $ python -c "import deep_ep"
  RuntimeWarning: Automatic NUMA balancing is enabled (kernel.numa_balancing=1), which can add ...
  

• Silent when kernel.numa_balancing=0:

  $ sudo sysctl -w kernel.numa_balancing=0
  $ python -c "import deep_ep"
  (no warning)
  

• Silent when EP_SUPPRESS_NUMA_CHECK=1:

  $ EP_SUPPRESS_NUMA_CHECK=1 python -c "import deep_ep"
  (no warning)
  

• Silent on non-Linux (no /proc/sys/kernel/numa_balancing): OSError caught silently.
• python -m py_compile deep_ep/__init__.py passes; git diff --check clean; file is excluded from the repository ruff target so no lint conflicts.

[alpha-baby]

[root@gpu001 ~]# cat /proc/sys/kernel/numa_balancing
1
[root@gpu001 ~]# uname -r
5.10.134-19.103.al8.x86_64
[root@gpu001 ~]# ps -p 347328 -o pid,ppid,comm,args
   PID   PPID COMMAND         COMMAND
347328 241159 python          /usr/bin/python -u /root/workdir/commands/__init__.py run-pipe --config model_config.yaml


[root@gpu001 ~]# ./funclatency-bpfcc -d 60 -m -p 347328 task_numa_work
Tracing 1 functions for "task_numa_work"... Hit Ctrl-C to end.

     msecs               : count     distribution
         0 -> 1          : 4        |****************************************|

avg = 0 msecs, total: 1 msecs, count: 4

Detaching...

I tested on my node with:

funclatency-bpfcc -d 60 -m -p 347328 task_numa_work

Result:

0 -> 1 ms : 4
avg = 0 msecs, total: 1 msecs, count: 4

So on this run I did not reproduce the high task_numa_work overhead from #624. The overhead was negligible for this PID/workload.
kernel.numa_balancing=, kernel=<uname -r>.

[toffee-desuwa]

Thanks for actually running the numbers, @alpha-baby. A real measurement beats hand-waving, so this is a useful data point.

One thing to clear up: the 16+ms figure isn't mine — it's from the trace in #624 (different kernel and workload from yours). The warning says "can add up to" on purpose; it's not claiming NUMA balancing is always slow. task_numa_work only costs anything when a process actually triggers NUMA page scanning, so it's very workload- and topology-dependent. The warning is based on that report rather than my own GPU runs.

Your 4 hits in 60s vs #624's 139 is about a 35x gap, which usually means the process just wasn't doing much cross-NUMA scanning during the trace. Two questions so we're comparing the same thing:

Was your run-pipe process actually doing DeepEP internode/intranode dispatch while you traced it? #624's overhead is specific to that comm hot path — if it wasn't, few task_numa_work hits is exactly what you'd expect.

What's the NUMA topology (sockets, does the model span NUMA nodes, cores pinned)? When memory doesn't cross NUMA boundaries, task_numa_work has nothing to do.

If overhead stays negligible even under a real internode-dispatch workload, that's genuinely worth knowing — it'd mean the warning should be narrowed (e.g. gated to specific kernels) rather than shown broadly.

Happy to adjust based on what you find.

[alpha-baby]

During the monitoring period of funclatency-bpfcc, DeepEP internode was being called frequently. We did not pin the LLM training processes to specific NUMA nodes.

[toffee-desuwa]

That answers both questions clearly — thanks. So: real internode dispatch, no NUMA pinning, and still negligible overhead. That's a strong counter-example.

The remaining variable is kernel version: you're on 5.10, while #624 was on 5.15 (NUMA balancing behavior changed meaningfully around 5.12+). But regardless of root cause, a warning that fires on setups where the overhead is near-zero isn't useful.

I'll narrow the check — likely gate it on kernel version or downgrade to a one-time info log instead of a warning. Will push an update.

[toffee-desuwa]

@alpha-baby Pushed an update based on your findings — thanks again for the data. Dropped the "16+ ms" claim, now says "milliseconds of tail latency on some kernel versions and workloads", and changed "Disable" to "If you observe unexpected latency spikes, consider:". Also kept the README consistent. Trigger logic and warning level unchanged.

[wenjianhn]

This issue is not a low hanging fruit for Claude Opus.
Given it doesn't understand the root cause.

[toffee-desuwa]

@wenjianhn Fair point. The check is just reading a sysctl flag — pretty surface-level. You wrote the original trace in #624, so you probably know what actually causes the overhead.
Any idea why @alpha-baby saw near-zero impact on a similar setup? Happy to rework the check if there's a better condition to trigger on.

[alpha-baby]

@wenjianhn Fair point. The check is just reading a sysctl flag — pretty surface-level. You wrote the original trace in #624, so you probably know what actually causes the overhead. Any idea why @alpha-baby saw near-zero impact on a similar setup? Happy to rework the check if there's a better condition to trigger on.

Sorry, I don't know why I can't reproduce this problem in my environment.

[wenjianhn]

See #624 (comment)

[toffee-desuwa]

Thanks, that explains it. So the real variables are memory size and thread busyness, not kernel version — the kernel hands the scan work to whichever thread is busy, and bigger memory means longer page-table walks. That's also probably why @alpha-baby didn't hit it. My kernel-version guess was wrong.

I'll update the warning text and README to say this, instead of the vague "some kernel versions and workloads".

[toffee-desuwa]

Pushed d389e22 — the warning and README now describe the busy-thread + memory-size mechanism instead of the kernel-version guess. Thanks @wenjianhn.