Finding the Straggler Rank
Every rank waits for the slowest one, so a single degraded GPU taxes the entire cluster at full scale. The job is not detecting that a straggler exists — it is telling a slow GPU from a slow link from a slow dataloader, which the obvious metrics cannot do.
Why this is worth real effort
Meta's published analysis of their research supercluster gives the asymmetry plainly: infrastructure failures affected 0.2% of jobs but 18.7% of runtime. The largest attributed hardware category was InfiniBand links.
The same work reports mean time to failure collapsing with scale — from tens of days at 8 GPUs to hours at tens of thousands — and that identifying persistently underperforming "lemon" nodes cut large-job failures from 14% to 4%. Straggler hunting is not tuning. At scale it is the difference between a cluster that finishes runs and one that does not.
Start with the free signal
Before instrumenting anything: a PyTorch watchdog timeout already prints, per rank, how many NCCL operations were enqueued versus completed. The gap is that rank's stall depth, and the rank where the two are equal is the one everyone else is blocked on.
It costs nothing and it is already in the log you have. See the hang that isn't your network for how to read the rest of that output.
The discriminator: correlate, don't rank
A list of ranks sorted by step time tells you who and never why. What separates causes is looking at slowness alongside the physical telemetry from the same rank at the same moment.
| Slow, and also… | Cause |
|---|---|
| Clocks low, temperature high | GPU thermal throttling — cooling or placement, not the fabric |
| Clocks and temperature normal, communication time high | Network. A degraded link, or traffic rerouted onto a slower path |
| Utilisation low, data-wait time high | Dataloader or storage — the GPU is idle, not slow |
| Power draw notably below peers at equal utilisation | Power capping, or a genuinely underperforming device |
Megatron-LM's built-in straggler detector is designed around exactly this correlation — it reports per-rank round-trip time next to GPU power, temperature, utilisation, clock and estimated throughput, so the discriminating comparison is available in one line rather than assembled from four dashboards. Its documentation is explicit that the power and temperature columns exist to narrow a straggler down to a specific GPU when the cause is hardware.
Relative scoring beats thresholds
Absolute thresholds age badly across model changes and cluster generations. NVIDIA's resiliency tooling scores each rank against the fastest rank instead — a relative score of 0.5 means that rank is running at half the speed of the best one, which is interpretable without knowing anything about the workload.
Two scores are worth tracking separately: performance relative to peers, which catches one bad node, and performance relative to that rank's own history, which catches gradual degradation affecting everything at once — the case peer comparison is blind to by construction.
Note that this class of check involves inter-rank synchronisation, so run it periodically — every few minutes — rather than every step.
The order that saves the most time
- Read enqueued-versus-completed counts from the timeout output. Free, already there.
- Check whether the slow rank is missing from collectives or late to them. Missing is software; late is hardware or fabric.
- Pull clocks, temperature, power and utilisation for that rank at that moment. The table above resolves most cases here.
- Only if telemetry is clean, suspect the link — and then measure that pair specifically rather than the cluster average.
Related
- Why most published bandwidth tables are wrong — measuring per pair, not in aggregate
- Link flap — when the straggler is intermittent and moves
- AI cluster networking — the full symptom index