Repository navigation
Sandbox: .source broken by nnsight bump — SourceEnvoy gained required args (6 remote tests) #281
Description
Rebuilding the image picks up nnsight's 0.8 branch tip (requirements.txt pins the branch, not a commit). The current tip breaks .source on the untrusted path.
Symptom
tests/test_nnsight_remote.py against a live deployment, untrusted path:
6 TypeError: SourceEnvoy.__init__() missing 2 required positional arguments: 'source' and 'line'
2 AttributeError: 'NoneType' object has no attribute 'modules' (TestRemoteCache)
2 OutOfOrderError: '...grad.i0' (TestRemoteGradients — known, documented)
1 TestRemotePeft::test_adapter_applied_remotely (fails on the trusted path too — separate)
11 failed, 40 passed
The trusted path is 50 passed, 1 failed (only the peft one). docs/developing/testing.md records the untrusted baseline as "46 passed, 2 failed, 2 skipped", so the source and cache failures are new.
Cause (source)
sandbox/nns.py:313 constructs a SourceEnvoy with three arguments:
return SourceEnvoy(self._envoy, name, f"{self._prefix}.{name}")SourceEnvoy.__init__ takes five — envoy, name, path, source, line. This worked because source and line had defaults at the commit ndif used to pin:
# nnsight ad939e9 (the old pin)
source: str = "",
line: int | None = None,nnsight b16ca0c3 ("vLLM: CUDA-graph taps, controller-only handoff, one request record, TP/DCP fixes") removed the defaults, making both required. The signature is otherwise unchanged.
IPCSource already round-trips a SOURCE event to the host, which source-instruments the real module and returns the operation names. The fix is presumably to return source/line alongside those names and pass them through — the host has them; the runner does not.
Cause (cache) — less certain
The two TestRemoteCache failures are AttributeError: 'NoneType' object has no attribute 'modules'. nnsight's cache.py also changed in this range. I have not traced this one to a specific commit; flagging it as part of the same bump rather than claiming the mechanism.
Broader point
This is the second time a branch-tip nnsight bump has silently broken the sandbox — the first was Mediator.pending becoming a Pending NamedTuple. Both were invisible until the untrusted path was exercised, because the trusted path shares no code with the runner shim. Worth considering either pinning nnsight to a commit again (the comment in requirements.txt argues for it) or running tests/test_nnsight_remote.py -p conftest_untrusted in CI against a local stack.
Observed on the Test deployment. ndif 0.8 @ be38d78, nnsight 0.8 @ 3bed88cc (previously 5be282b6, which passed).
Activity
JadenFiotto-Kaufman commentedon Sep 9, 2026
Fixed in
413ba0b. Also correcting my attribution above: only the.sourcehalf is from the nnsight bump. I flagged the cache failures as "part of the same bump" with a caveat, and having traced them, that was wrong..source— was the bumpAs diagnosed.
install_sourcenow returns the operation names together with the forward's source text and each operation's line, andIPCSourcepasses them through toSourceEnvoy.Passing empty stubs would have been enough to restore tracing, since
source/lineonly feed__repr__— but the host already builds theCompiledthat a localSourcereads them off, so handing them over costs nothing and gives the untrusted path the same repr as the trusted one instead of the dotted-path fallback.Compiled.codestays on the host; it is a code object and does not pickle.tracer.cache()— was not the bumpShippingCachepassedNoneas its model:Cache.subscriptionsreads every module's path off the model when the cache names no explicit targets — exactly what a baretracer.cache()asks for — soInterleaver.__enter__died with'NoneType' object has no attribute 'modules'.subscriptionsand__getstate__are byte-identical across5be282b6..3bed88cc, so this is a latent bug that needed a cache with no module list to surface, not a regression. The host has the model; it is now passed in.Result
The remaining three are the two
TestRemoteGradients(the known sandbox limitation recorded indeveloping/testing.md) andTestRemotePeft, which fails on the trusted path too — #282.One aside worth recording:
TestRemoteEarlyStop::test_stopfailed on the first full run after the fix and passed on the second, and passes 3/3 in isolation. It looks flaky under the full-suite run rather than related to either change.The broader point in the original report stands — this is the second bump to break the sandbox silently, and the untrusted suite is what caught both.
sandbox: give the runner a real SourceEnvoy, and the host cache a model
JadenFiotto-Kaufman commentedon Oct 5, 2026
Update on the Test image (nnsight 0.8.0rc1): one-level
.sourceis fixed — theSourceEnvoy.__init__TypeError no longer occurs. Two related failures remain:.sourceis still broken on the untrusted path.inner = h[0].attn.source.attention_interface_1.source; inner.<op>.output→SourceNotAvailable: recursive .source is only available inside a trace. Trusted/local OK. LikelySourceEnvoy.source(nnsightsource.py:810) checksinterleaver.interleaving, which is never true in the runner, andIPCSource(sandbox/nns.py:286) has no recursive path. This breaks the documented route to eager attention probabilities (attention_interface_1.source.attn_weights_2).TestRemoteCache'NoneType' ... modulesfailure has a client-side sibling. Navigating a remote cache by attribute (cache.transformer.h[5]) rather than by path key (cache["model.transformer.h.5"]) raisesAttributeError: 'NoneType' object has no attribute 'path'at nnsightintervention/cache.py:338—self._cache.modelis None after deserialization. Path-key access works; attribute navigation does not. Same cache-deserialization root; the fix is probably to rebindcache.modelto the client envoy when results land. (This piece is an nnsight-client bug and could be split to the nnsight repo if preferred.)JadenFiotto-Kaufman commentedon Oct 13, 2026
The recursive-
.sourcehalf of this issue is fixed by #307, now merged intodev— nested drills (...attn.source.attention_interface_1.source.<op>) work on the sandboxed path with values bit-identical to trusted; missing-op messages match local byte-for-byte. Small follow-ups tracked separately.Leaving this open for the remaining bullet: the client-side cache attribute-navigation failure (
cache.transformer.h[5]→AttributeError: 'NoneType' object has no attribute 'path';cache._cache.modelis None after deserialization — path-key access works). That one belongs in the nnsight repo.