The Hang That Isn't Your Network
PyTorch has two separate mechanisms that abort a stuck distributed job, and they mean opposite things. Confusing them sends teams to inspect a fabric that was never involved.
Two watchdogs, not one
The watchdog tracks whether collectives complete. When one exceeds its timeout, the watchdog aborts and reports which collective stalled. That is a genuine distributed-communication event, and the network is a reasonable suspect.
The heartbeat monitor watches the watchdog itself. PyTorch documents TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC as controlling "the watchdog heartbeat timeout period after which the monitoring thread will abort the process."
Read that carefully. If the monitoring thread fires, it did not observe a slow collective. It observed that the thread whose job is to observe collectives stopped responding. Something is blocking the process itself.
The distinguishing symptom
A heartbeat abort names a CUDA API call or the watchdog being stuck — not a collective, not a rank, not a device. The usual causes are a hung CUDA or driver call, or another Python thread holding the GIL inside a CUDA API.
If that is what you are looking at, stop investigating the network. No amount of fabric tuning fixes a GIL deadlock.
The one number worth extracting
A watchdog timeout prints, per rank, the count of enqueued versus completed NCCL work. The difference is that rank's stall depth — and the rank where the two numbers are equal is the one everybody else is waiting for.
That is the cheapest straggler signal available and it requires no instrumentation. Most people scroll past it because it sits below the stack trace.
Turn the flight recorder on before you need it
PyTorch's flight recorder keeps a ring buffer of collective events, which is what lets you reconstruct a hang after the fact. Two variables matter and they are coupled:
TORCH_NCCL_TRACE_BUFFER_SIZE— documented as "the maximum number of events we store in the flight recorder's ring buffer." Set it explicitly; a recorder with no buffer records nothing.TORCH_NCCL_DUMP_ON_TIMEOUT— "control whether dumping debug info on watchdog timeout or exception is detected." Depends on the buffer above being sized.
The underused one is TORCH_NCCL_DEBUG_INFO_PIPE_FILE — "the pipe file to trigger debugging dump manually, write anything into the pipe would trigger the dump." That dumps a live, still-hung job without killing it, which is exactly what you want when a job has been stuck for twenty minutes and you have one shot at understanding it.
Also worth setting before a long run: TORCH_NCCL_ENABLE_TIMING ("compute accurate collective timing per-collective") and TORCH_NCCL_DESYNC_DEBUG ("helpful in figuring out the culprit rank of collective desync").
Check the defaults against your own PyTorch version — they have changed between releases, and the current documentation does not state all of them.
Deciding in thirty seconds
- Message names a stuck collective → distributed problem. Check the fabric.
- Message names the watchdog being stuck or a CUDA API → process problem. Check the driver and your Python threads.
- A rank missing entirely from a sequence → it never reached the collective. Software, not network.
Related
- AI cluster networking — the full symptom index
- How to read a NCCL error log — when it is the fabric