How to Read a NCCL Error Log
A failed collective produces hundreds of lines and roughly one of them is the cause. Here is which one, and how to tell within seconds whether you are looking at a fabric problem, a node problem, or something that was never the network at all.
The one rule that matters
status=5 is never the cause. It is IBV_WC_WR_FLUSH_ERR, and it carries vendor err 249 on Mellanox hardware. Once a queue pair enters ERROR state, every work request still posted to it flushes and logs one of these.
So a real failure looks like one line with a genuine error, followed by hundreds of 5/249 lines. People paste the end of the log — which is all flush noise — and debug the wrong thing for a day. Scroll up to the first completion error and ignore everything after it.
Decoding the completion line
The line you are looking for is generated by this format string in src/transport/net_ib.cc:
NET/IB : Got completion from peer <addr> with status=<N> opcode=<N> len=<N> vendor err <N> (<type>)The status field is a standard verbs completion code:
| status | Meaning | Usually |
|---|---|---|
| 4 | LOC_PROT_ERR | Memory registration or permissions |
| 5 | WR_FLUSH_ERR | Collateral damage. Not a cause. |
| 9 / 10 / 11 | REM_INV_REQ / REM_ACCESS / REM_OP | The far side rejected the operation |
| 12 | RETRY_EXC_ERR | The far side stopped answering — link, congestion, or a dead peer |
| 13 | RNR_RETRY_EXC_ERR | Receiver had no buffer posted, repeatedly |
status=12 vendor err 129 above a wall of 5/249 is the single most common real failure. It means the peer stopped acknowledging.
Two free classifiers hiding in that line
RoCE or InfiniBand, without asking
NCCL appends localGid and remoteGids to the completion line only when the port's link layer is Ethernet. In the source the whole block is guarded by if (… link_layer == IBV_LINK_LAYER_ETHERNET).
So: GID strings present means RoCE. Absent means native InfiniBand. No need to ask what someone is running — the log already said.
Which NCCL they are actually running
Upstream NCCL 2.19 and later says status=12. Older builds and the out-of-tree AWS plugin say with error 12 instead. If a log says "error" rather than "status", you are debugging a different code path than the one you have open.
"Remote process exited" when nothing exited
The completion-error branch returns ncclRemoteError directly. That value propagates up and surfaces as a message about a remote rank dying — which sends people hunting for an OOM kill or a crashed process that never happened.
A transport error and a dead peer produce the same user-facing string. When you see it, go find the completion line before you go find a corpse.
The highest-signal line in the whole log
This one is worth grepping for first, because it is unambiguous:
NCCL WARN NET/IB : mlx5_4:1 Got async event : port error The text comes from ibv_event_type_str() — port error, port active, IB device fatal error, local work queue catastrophic error, and similar. (IBV_EVENT_COMM_EST is deliberately suppressed, so anything you see here is real.)
What makes it valuable is where it comes from: a dedicated polling thread, not the collective. It is out-of-band, and it typically lands seconds before the watchdog timeout on the same host. That makes it an excellent correlation key — find the async event, and you have found both the moment and the device.
When it is the node, not the network
A family of errors comes from the verbs wrapper and all share the shape Call to <fn> failed with error <strerror>. The one you will actually meet:
Call to ibv_reg_mr failed with error Cannot allocate memory That is not memory pressure. It is the memlock ulimit, and it is extremely common in containers, where the limit does not inherit the way people expect. Set * soft memlock unlimited and * hard memlock unlimited in limits.conf, pass --ulimit memlock=-1 to Docker, or set PropagateResourceLimits under Slurm.
Its neighbour, Call to ibv_modify_qp failed with error Invalid argument, points at a GID index or RoCE configuration mismatch rather than at memory.
The three lines you want to see
Run with NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,ENV,NET,GRAPH. Their absence tells you more than most metrics.
Using network IB
If this says Socket, NCCL fell back to TCP and nothing else in this article applies.
GPU Direct RDMA Enabled for GPU 0xNN / HCA 0xMM (distance 2 <= 3), read 1
The GPU reaches the NIC directly. The "Disabled" variant helpfully prints both the measured PCIe distance and the threshold it exceeded.
Channel 00/0 : 3[3] -> 11[3] [send] via NET/IB/2/GDRDMA
Per-channel confirmation. Missing /GDRDMA on inter-node channels is roughly a halving on its own.
Triage order
- Grep for
Got async event. If present, you have the device and the moment. - Find the first completion error. Ignore every
status=5. - Check for GID strings to know whether you are on RoCE or InfiniBand.
- Confirm
Using network IBandGPU Direct RDMA Enabled. - Only then start changing things.
Related
- AI cluster networking — symptom index for the rest of the fabric
- Why your GPU fabric runs at half speed — when the log is clean and the bandwidth is not
- TORswitch — 32-port 100G AI network switch