Link Flap: The Failure That Never Reports a Failure

What it looks like

Step time is stable, then one iteration takes several seconds longer than the rest. No rank crashed. No error surfaced. By the time anyone looks, the link is up and the counters look ordinary. It repeats at no particular interval.

A link flap is a physical layer transition — the link drops and re-establishes fast enough that monitoring polling at 30-second intervals never sees it. But a collective operation notices immediately, because every rank waits for the slowest one.

How often this actually happens

Alibaba Cloud published operational figures for their LLM training fabric in Alibaba HPN (SIGCOMM '24). From their production clusters:

"0.057% of NIC-ToR links fail each month, and about 0.051% of ToR switches encounter critical errors and crashes. Under this high failure rate, a single LLM training job would encounter 1-2 crashes each month. Furthermore, 5K-60K link flapping cases happen each day, introducing temporary performance degradation as well."

Those percentages look small until you multiply them. As the paper notes, a single LLM training job involves "dozens of ToRs and thousands of optical modules and links." A 0.057% monthly failure rate across thousands of links is a near-certainty of something failing during any long run — and flapping is an order of magnitude more common than outright failure.

What one flap costs

The same paper measures it directly. Training LLaMa-7B on 256 GPUs across 32 hosts, with link malfunctions injected on a NIC-ToR link:

"In single-ToR, the temporary link flapping halts the training for more than nine seconds. In dual-ToR, the performance degradation is negligible."

Nine seconds, from a transition too brief for monitoring to record. And the comparison in the outright-failure case is just as stark — a single-ToR link failure halts training immediately, while the dual-ToR topology absorbs the same failure at 6.25% degradation and returns to full throughput the moment it is repaired.

Finding it

Because the link is healthy by the time you look, you diagnose flap from history rather than from state.

• Link-down counters, not link state. A port whose carrier-transition count climbs while its status stays "up" is the whole signature.
• Correlate against NCCL async events. The Got async event : port error line is emitted by a dedicated polling thread, out of band from the collective, and usually lands seconds before the stall registers anywhere else. It names the device and the moment — see how to read a NCCL error log.
• Read the optics, not just the port.ethtool -m <iface> exposes per-module temperature, supply voltage, bias current and TX/RX optical power. Receive power drifting toward the low end of the module's spec is a link that will flap before it fails.
• Watch corrected FEC counts. A link correcting errors is a link with no margin left. Uncorrectable blocks are the failure; a rising corrected count is the warning that precedes it.

The unglamorous causes

Before suspecting the module, rule out the two things that cause most marginal links in practice: contaminated connector end-faces and bend radius violations. A single dust particle on a fibre end-face attenuates enough to eat a link's optical budget, and it transfers between connectors every time someone mates them. Inspect and clean before replacing.

After that: receive power near the low limit, temperature at the top of the operating range, and modules running outside their specified reach. Marginal is the operative word — a link that is out of budget fails cleanly and gets fixed. A link with 0.5 dB of margin flaps for months.

Related

• AI cluster networking — transceivers by rate, and the rest of the symptom index
• Why your GPU fabric runs at half speed — the other topology problem you cannot tune away
• L11 rack integration — topology, cabling and optics specified together