Cluster fabric
AI cluster networking
Most GPU cluster performance problems are fabric problems wearing a different costume. This page starts from what you are seeing and works back to what it is — then to the switching, optics and rack topology that resolve it.
Start from the symptom
Nine problems account for most of what goes wrong on an AI fabric. Three of them are not network problems at all, which is worth knowing before you spend a week on the network. Rows without a link are answered in the row itself.
| What you see | What it usually is | |
|---|---|---|
| Roughly half the expected bandwidth, every link healthy | ACS routing traffic through the CPU root complex, or routing collisions on per-rack cabling | Diagnose → |
| NCCL reports a remote rank died, but nothing died | A queue pair entered ERROR state; the real cause is buried above hundreds of flush errors | Diagnose → |
| Job hangs with no error, aborts after 480 seconds | Heartbeat monitor, not the watchdog — a CUDA or driver hang, or a GIL deadlock. Not the network | Diagnose → |
| Bootstrap cannot find a network interface | SR-IOV virtual functions and secondary physical functions beside the routable one; NCCL picks a non-routable interface | Diagnose → |
| Fine at small message sizes, collapses at large ones | Incast against shallow switch buffers. Many senders, one receiver, simultaneously — check ECN marking thresholds before adding PFC, which spreads backpressure into victim flows | — |
| Intermittent stalls, no link ever reports down | Link flap. At scale these run to thousands of events per day, and a single flap can halt training for seconds | Diagnose → |
| One rank is consistently last | A straggler — GPU thermal throttling, a degraded link rerouting through NVLink, or the dataloader | Diagnose → |
| Completion errors only on messages over 1024 bytes | MTU mismatch. Host at 1500 instead of 9000, or a switch below it | — |
| A third-party module links up, then flaps intermittently | Its EEPROM vendor string matched nothing in the media-settings table, so the port fell back to generic SerDes tuning | Read → |
What a healthy fabric looks like
Run a collective with NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,ENV,NET,GRAPH and check for these three lines. Their absence is more informative than most metrics.
Using network IB
InfiniBand or RoCE is engaged. If this says Socket, NCCL fell back to TCP and everything below is moot.
GPU Direct RDMA Enabled for GPU 0xNN / HCA 0xMM
The GPU talks to the NIC directly. "Disabled" names the PCIe distance and the threshold it exceeded.
via NET/IB/2/GDRDMA
Per-channel confirmation. Missing /GDRDMA on inter-node channels is roughly a halving on its own.
Node-side checks
sudo lspci -vvv | grep -i acsctl— must showSrcValid-. ASrcValid+means ACS is on and GPUDirect is degraded.- Host MTU 9000, switch MTU at or above it. Completion errors that appear only on messages larger than 1024 bytes are an MTU mismatch.
numa_balancingset to 0.
Topology is the part you cannot tune
Node settings are recoverable. Cabling is not. Routing collisions — where two flows land on the same link somewhere in the path — can halve throughput on an otherwise perfect cluster, and they are a property of how the fabric is wired rather than how it is configured.
They do not occur on a single switch, and they do not occur on a rail-optimized topology. They are common on classic per-rack cabling. That makes topology a purchasing decision, made when the rack is specified, not a tuning exercise afterwards.
Switching
TORswitch
32-port 100G AI network switch. Specified against the topology rather than picked afterwards — see L11 rack integration for cabling, optics and cluster validation before it ships.
Transceivers
Reach, coding and connector are matched to your switches; exact configuration is confirmed at quote. Grouped by rate — pick the row that matches your fabric.
Site-to-site optical transport
A different problem from cluster fabric. The F520 line is DWDM and OTN equipment for connecting sites — data centre interconnect, metro and long-haul — not for wiring GPUs to each other. It matters when your training data lives in one facility and your cluster lives in another.
Specifying a cluster?
Topology, switching and optics are decided together. TORmem quotes them as one integration rather than as a parts list.