Repository navigation
TestRemotePeft::test_adapter_applied_remotely fails on both paths — architecture mismatch on the adapted model #282
Description
The per-request PEFT adapter test fails identically on the trusted and untrusted paths, so unlike #281 this is not sandbox-specific.
Symptom
tests/test_nnsight_remote.py::TestRemotePeft::test_adapter_applied_remotely
RemoteError: The model architecture on this server doesn't match the one your code was
traced against: it has no module at 'Module:model.base_model.model'.
TestRemotePeft::test_env_reports_adapter passes — the adapter id does ride on the request env. It is applying it, or the module tree it produces, that does not line up.
The client builds a meta model with peft grafted (so it can name base_model.model.*); the server is expected to apply the adapter before the run so those paths resolve. The mismatch says the server's live tree has no base_model.model, i.e. the adapted architecture is not what the client assumed.
Notes
- Client and server ran the same image here, so this is not the usual version-skew cause the message suggests.
peft==0.19.1,transformers==5.16.1, nnsight3bed88cc. - Reproduces on both key types (trusted and untrusted), 1 failure in each run.
- The error message itself is behaving correctly — it is the architecture-mismatch classification doing its job, which is how the cause was this easy to read.
Observed on the Test deployment. ndif 0.8 @ be38d78.
Activity
JadenFiotto-Kaufman commentedon Sep 9, 2026
Root-caused, and it is not an ndif bug — filed upstream as ndif-team/nnsight#726.
The ndif side is correct: the client's envelope carries
"env": {"peft": "peft-internal-testing/gpt2-lora-random"}(verified on the wire),validate_requestkeeps it, and the actor applies it with_remoteable_set_envbeforeexecuteon both paths.The problem is that
_remoteable_set_envwraps the torch module but does not rebuild the envoy tree around it, so the paths the client traced against never exist server-side:h[0]TransformersModel(REPO, peft=ADAPTER)— what the client doesmodel.base_model.model.transformer.h.0_remoteable_set_env({"peft": ADAPTER})— what the server doesmodel.transformer.h.0After the rebind the module tree is right (286 modules,
base_model.model.*present) butbase_modelhas no child envoys, and the pre-wraptransformerenvoy still resolves — so the lookup goes through the stale branch and reports a path withoutbase_model.modelin it.Reproducible with nnsight alone, no server involved; full repro in the upstream issue.
This also explains the 138 ms failure and the absence of any peft/adapter/lora activity in the actor logs: the deserialization fails before the adapter is ever exercised.
Nothing to fix here — this can close when nnsight#726 lands, or stay open as the downstream tracker.