When NCCL Picks the Wrong Network Interface
The interface exists. It is up. It has the node's IP. And bootstrap still fails, or succeeds at a fraction of the bandwidth. On modern GPU nodes the cause has moved, and the advice you will find online is a generation out of date.
NCCL falls through on purpose
When no interface is specified, NCCL walks a fallback chain: InfiniBand interfaces first, then a subnet hint derived from the communicator ID, then everything except docker and loopback, then docker, then loopback.
That last step is deliberate. NCCL would rather bootstrap over lo than refuse to start — which is why the classic symptom was a job that ran, slowly, over docker0.
The modern failure is SR-IOV, not Docker
A current GPU node presents far more interfaces than it used to: the routable physical function, a second physical function that is cabled but has no address, and a row of SR-IOV virtual functions that also have no addresses. NCCL cannot tell which of these is the one you meant, and a filter that was correct in one region can be wrong in another where the naming differs.
Two public postmortems from a Stanford project running on CoreWeave H100 nodes show both halves of the trap:
- Too narrow. An exact-match filter using the
=prefix matched nothing on that build — the interface existed, was up, had the node IP, and the environment variable was verified as injected. It surfaced asBootstrap : no socket interface found. - Too broad. The exclude-list that fixed it was then insufficient in a different region, where a second PF and eight SR-IOV VFs sat beside the routable interface — NCCL picked a non-routable one and bootstrap failed again.
The lesson is not a magic filter string. It is that any static interface filter is a guess about node topology, and it silently stops being true when you move region, image, or instance type.
Filter syntax, since it trips people
| Form | Meaning |
|---|---|
| eth | Prefix match — every interface starting "eth" |
| =eth0 | Exact match. Verify it works on your build before relying on it |
| ^docker,lo | Exclude — everything except these prefixes |
What to do instead of guessing
Derive the interface from the routing table at launch and pin it, rather than hard-coding a name or maintaining an exclude list that rots. The interface carrying the default route is the one you want, and it stays correct across regions and images.
Then confirm from the log rather than assuming. NCCL prints which interface and address the out-of-band plane chose, and separately whether IB engaged at all — if you see Using network Socket where you expected Using network IB, the selection already went wrong and everything downstream is moot.
Related
- How to read a NCCL error log — the lines that confirm selection went right
- AI cluster networking — the full symptom index