Programmable Silicon in the Network
AI fabrics stopped being dumb pipes. The question is no longer whether the network should do work, but whether that work should be frozen into an ASIC on a three-year cycle — while the algorithms it implements are still being argued about in papers.
Why the network got smart
A training job's communication pattern is nothing like general cloud traffic. It is a small number of enormous, synchronised, all-to-all bursts, repeated forever. Every rank waits for the slowest, so the fabric does not degrade gracefully under contention — it converts a local hotspot into cluster-wide idle time.
That changed what a switch is for. Three jobs migrated into the network:
- Reduction. If switches can sum partial results as they pass through, an all-reduce stops being a ring of point-to-point transfers. NVIDIA's SHARP is the fixed-function version of this idea, and it is why bus-bandwidth arithmetic stops working when it is enabled.
- Load balancing. ECMP hashes flows onto paths. With few, long, fat flows it hashes badly and two collectives collide on one link.
- Congestion control. Deciding how fast to send, using signals the switch generates — ECN marks, measured queue occupancy, in-band telemetry.
The problem with freezing that into silicon
Fixed-function ASICs have scaled capacity impressively — Alibaba's HPN work describes building on a 51.2 Tbps single-chip switch, and charts the generational climb from 6.4T through 12.8T and 25.6T to get there. Capacity is not the constraint.
Behaviour is. The algorithms these chips implement are not settled science. The same paper, surveying the published work on ECMP load imbalance, dismisses much of it as impractical or "unverified in large-scale deployment" — and that is a hyperscaler with production LLM clusters describing the current state of the art, not a vendor being dismissive.
Their own measurements show what is at stake. Under a conventional Clos topology, imbalanced load kept switch queue lengths at 267 KB and 3 KB on the two paths toward the same NIC — during a real GPT-3-class production job. After changing the topology, the average queue fell to 20 KB. Same hardware, same traffic; the difference was a design decision about how traffic is placed.
An ASIC commits to answers like that at tape-out. If the field's understanding of congestion control or path selection moves in the following two years — and it has, repeatedly — the silicon cannot follow.
What programmable silicon actually buys
The case for an FPGA in the data path is not that it is faster than an ASIC. It is not. It is that the function it performs can change after the hardware ships.
| Fixed-function ASIC | Programmable / FPGA | |
|---|---|---|
| Throughput ceiling | Highest available | Lower per watt and per dollar |
| Power | Optimised for its one job | Higher for equivalent work |
| Time to change behaviour | A silicon respin — years | A bitstream — weeks |
| New protocol or algorithm | Next generation, if ever | Deployable to installed hardware |
| Economics | Wins at hyperscale volume | Wins below it |
That last row is the one people skip. Taping out an ASIC only makes sense across enormous volume. Below that threshold, programmable logic is not a compromise chosen for lack of resources — it is the correct engineering answer, and it is why in-network research keeps being demonstrated on programmable data planes rather than on custom chips.
The question to ask a switch vendor
Not "how many terabits." Ask: when the congestion control story changes in eighteen months, what happens to the hardware I bought?
For a fixed-function switch the honest answer is "nothing — you buy the next one." That may be perfectly acceptable at hyperscale refresh rates. It is a different proposition for an enterprise or NeoCloud operator expecting five years out of a fabric.
Where TORmem sits
TORswitch is a 32-port 100G switch built with custom FPGA acceleration in the data path, which is a deliberate choice about that trade rather than a component selection.
We are not going to publish performance claims for it in a general article about the category — the specific functions, the measured numbers, and what they beat belong on the product page with data attached, not in prose. If you are evaluating it, ask us directly and we will tell you what it does and what it does not.
The broader point stands regardless of whose switch you buy: in a fabric whose bottleneck is collective communication, the silicon's flexibility is a specification, not a footnote.
Related
- Why most published bandwidth tables are wrong — including what in-network reduction does to the arithmetic
- Why your GPU fabric runs at half speed — what path collisions cost in practice
- AI cluster networking — the full symptom index