Why Your GPU Fabric Runs at Half Speed
Two causes account for most cases where a cluster measures roughly half its expected bandwidth with every link reporting healthy. One is a node setting. The other is how the rack is cabled — and no amount of tuning fixes it.
The symptom
Bus bandwidth lands near 50% of what the hardware should deliver. Every link is up. ibstat is clean, error counters are zero, and a point-to-point test like ib_write_bw may even look fine. Only collectives are slow.
The tell that separates this from ordinary tuning problems is that it is pair-dependent. Some node pairs are fast, others are slow, and the same node can be fast with one partner and slow with another.
First, make sure you are measuring the fabric
Before diagnosing anything, confirm the benchmark is testing the network at all. A two-node all-reduce mostly measures NVLink, not your fabric. NCCL maintainer Sylvain Jeaugey, in nccl-tests #106:
"Can you run again with NCCL_ALGO=RING? By default, on two nodes, we use the Tree algorithm, which does not directly reflect the GPU-NIC speed, and that makes it harder to 'see' issues through the BW numbers."
This is also why published per-generation bandwidth tables are usually wrong: many are built from two-node runs that never stressed the fabric they claim to characterise.
The diagnostic that splits the two causes
There are two common causes, and one question separates them. Take the slow node and test it against several different partners:
- If no partner reaches full bandwidth — the problem is on that node. Go to cause 1.
- If some partner does reach full bandwidth — the node is fine and the fabric is not. Go to cause 2.
That test costs ten minutes and eliminates half the search space. It is the first thing worth doing and it is almost never the first thing done.
Cause 1 — ACS is enabled
PCIe Access Control Services forces peer-to-peer traffic up through the CPU root complex instead of letting it move directly between the GPU and the NIC. Jeaugey names the effect precisely: "ACS being enabled on a node, causing all traffic to be routed to the CPU root complex, thereby halving the performance."
Check it as a pass/fail string:
sudo lspci -vvv | grep -i "acsctl"SrcValid- means ACS is off, which is what you want. SrcValid+ means it is still enabled on that device and GPUDirect RDMA is silently degraded. Confirm the result in the NCCL log itself — with NCCL_DEBUG=INFO you want to see GPU Direct RDMA Enabled, and channels reporting via NET/IB/2/GDRDMA. The absence of /GDRDMA on inter-node channels is roughly a halving on its own.
If you are virtualised, this may not be fixable. ACS generally cannot be disabled under QEMU/KVM passthrough. One reported comparison put bare metal at roughly 320 GB/s and the same hardware in passthrough VMs at roughly 180 GB/s — a 44% regression with no configuration path out of it.
Cause 2 — routing collisions, which are a cabling decision
If the node is clean, the fabric is producing collisions. Jeaugey again, and this is the sentence worth internalising before you buy a rack:
"Some nodes may be lucky and they can have all their flows in each direction use different links throughout the network, while some other pairs may have flows using the same link at some point in the routing, causing a bottleneck. That could easily halve the performance. This does not happen on single switches, nor on rail-optimized topologies, but is frequent on classic per-rack network cabling."
That is not a tuning problem. It is a property of how the cluster is wired. The same issue thread shows what it looks like in production — identical hardware in the same data centre, measured pair by pair:
| Node pair | Bus bandwidth |
|---|---|
| gn632, gn641 | 88.36 GB/s |
| gn404, gn699 | 86.81 GB/s |
| gn644, gn699 | 49.36 GB/s |
| gn641, gn644 | 49.30 GB/s |
| gn284, gn678 | 49.28 GB/s |
Roughly half the pairs at roughly half the bandwidth. Note that crossing a top-of-rack switch was not the predictor — one of the fast pairs crossed a ToR and one of the slow pairs did not. That is the signature of collisions rather than of oversubscription.
One more thing worth checking on InfiniBand
In that same thread the reporter eventually found their service-level mask was set to 0xFFFE. Moving NCCL_IB_SL from 0 to 1 measurably improved the distribution — the share of node pairs reaching 70 GB/s or better rose from 13% to 20%. It did not solve the problem, but it is a cheap check and it is rarely documented.
What this means if you are still buying
Cause 1 is a setting and you can fix it this afternoon. Cause 2 is structural. Adaptive routing helps and is worth enabling, but the durable answer is topology: a single switch where the cluster fits inside one, and a rail-optimized fabric where it does not.
That decision is made when the rack is specified, not after it is racked. It is the single most expensive thing to get wrong and the cheapest thing to get right.
Related hardware
TORmem builds rail-optimized fabrics as part of full rack integration. The switching and optics are specified against the topology rather than picked from a catalogue afterwards.
- TORswitch — 32-port 100G AI network switch
- Full L11 AI rack integration — topology, cabling, optics and cluster validation before it ships
- CXL vs. RDMA over Ethernet — why TORmem builds on Ethernet and RDMA