GPU cluster networking: How optical transceivers impact AI training performance

Introduction

The rise of large scale AI models has changed what we expect from data center networks. Training models with billions or even trillions of parameters requires distributing the workload across hundreds or thousands of GPUs working together as a cluster. A GPU cluster is a tightly coordinated system where every GPU depends on every other GPU to exchange data, synchronize results, and advance through the training process together.

A single GPU, no matter how powerful, cannot train a modern large language model alone. The datasets and parameter counts are too large. This is why organizations deploy clusters, scaling from 8 GPUs in a single server to thousands of GPUs spread across an entire data center.

The challenge lies in connecting them effectively:

GPU cluster networking falls into two layers:

Within the server (intra-node), GPUs communicate through high-speed interconnects like PCIe and NVLink, which provides direct GPU-to-GPU links at up to 900 GB/s without routing through the CPU.

Between servers (inter-node), GPUs exchange data over the data center network using RDMA (Remote Direct Memory Access), which moves data directly between server memory, bypassing the CPU entirely. Two technologies enable this: InfiniBand, historically dominant but solely sourced through NVIDIA, and RDMA over Converged Ethernet (RoCEv2), which brings the same capability to standard Ethernet fabrics.

The inter-node network—the fabric connecting GPU servers to each other—determines whether a cluster runs at full efficiency or wastes expensive compute time waiting on data. The network design, the switches, the congestion management, and even the optical transceivers linking everything together all determine how much of the GPU capacity actually gets used for training versus sitting idle.

This blog examines how network bottlenecks arise in AI training clusters and what infrastructure decisions reduce GPU idle time from over 33 percent to below 15 percent.

Every byte of data exchanged between GPU servers crosses an optical transceiver.

“In a 1,000-GPU cluster, over 10,000 transceivers sit in the path of latency-sensitive training traffic.”

The quality, reliability, and consistency of these components directly determine whether the cluster runs at full efficiency or wastes GPU time on retransmissions, link failures, and unplanned downtime.

Network bottlenecks in AI training

The cost of wasted GPU time

Training a large AI model is expensive. Pre-training Meta’s Llama 2 model, for example, required between 184,000 GPU-hours for the 7-billion-parameter version and 1.7 million GPU-hours for the 70-billion-parameter version. With a single AI training server containing 8 GPUs costing upwards of $400,000, and GPUs accounting for up to 80 percent of total AI training costs, every minute of GPU idle time translates directly into wasted investment. When the network becomes a bottleneck, training shifts from being compute-bound to network-bound, meaning organizations are paying for expensive GPU capacity that sits waiting on data instead of processing it.

Why GPUs wait on the network

During distributed training, each GPU processes its portion of the data and computes adjustments called gradients. Before moving to the next step, every GPU must share its gradients with all others and combine them into a single unified update. This operation is called AllReduce: each GPU sends its gradients to the network, receives everyone else’s, and produces an identical combined result. The cluster moves forward only when every GPU has completed this exchange.

“Operation AllReduce is the heartbeat of distributed training—it runs at every training step, it touches every GPU, and it generates the bulk of the network traffic in the cluster. The entire cluster moves only as fast as its slowest link.”

In poorly designed networks, this synchronization overhead can consume 30 to 50 percent of total training time. A latency increase of just 100 microseconds can reduce overall training efficiency by up to 15 percent. When networking is optimized, GPU utilization can increase from the 65–75 percent range to above 90 percent, and training jobs complete faster. ResNet-50 training benchmarks have shown a reduction from 28 minutes to 18 minutes simply by improving the underlying network.

A critical concept here is tail latency, the condition where a small number of outlier workloads slow down the completion of the entire training job. Because all GPUs must synchronize before moving forward, even one GPU experiencing network delay holds back every other GPU in the cluster. Eliminating tail latency is key to maximizing job completion time and the return on GPU investment.

Where bottlenecks occur in the fabric

“Modern GPU NICs operate at 400 Gbps (with 800 Gbps adoption accelerating), so raw port bandwidth is rarely the issue. The bottlenecks arise from how the network fabric handles traffic at scale.”

Packet loss has severe consequences. Unlike traditional data center traffic that uses TCP with selective retransmission, GPU clusters communicate using RDMA over Converged Ethernet v2 (RoCEv2), which encapsulates data in UDP. When a single packet is lost, the RDMA layer cannot selectively recover it. Instead, it waits for a timeout and retransmits the entire RDMA operation or all packets after the lost one in the sequence. A single dropped packet out of hundreds forces a large retransmission, degrading performance and adding unnecessary load to the network.

Packet loss itself can stem from network congestion (when ingress traffic exceeds egress capacity and buffers overflow) and bit errors (caused by faulty cables, transceivers, loose connections, or temperature fluctuations; when errors exceed what Forward Error Correction can recover, frames fail CRC checks and are dropped).

Several other fabric-level problems compound the damage:

Elephant flows and load balancing failure. AI training produces a small number of very large, long-lived flows rather than thousands of small ones. Standard ECMP (Equal-Cost Multi-Path) routing hashes flows across parallel paths, but with few distinct flows each running at 400 Gbps, multiple elephant flows land on the same uplink while other paths sit idle. This creates congestion at two points: leaf-to-spine uplinks when ECMP assigns multiple flows to the same path, and spine-to-leaf downlinks when flows from several switches converge on the same destination.

Synchronized bursts and PFC (Priority-based Flow Control) cascading. GPUs complete computation at roughly the same time, then simultaneously flood the network with traffic. Buffers fill rapidly. PFC prevents packet loss by pausing upstream senders, but when one GPU NIC stalls, its pause frames propagate through the fabric and slow down completely unrelated GPU pairs sharing the same paths. One slow port cascades into a cluster-wide slowdown.

Single link failures halt everything. AllReduce requires every GPU to participate. One failed link stops the entire training job and loses GPU-hours until the fault is resolved.

The shift from InfiniBand to Ethernet

Historically, InfiniBand dominated AI back-end fabrics due to built-in RDMA support and low latency, but it is solely sourced through NVIDIA, making it expensive and supply-constrained. Today, Ethernet with RoCEv2 offers an alternative, delivering a diverse vendor ecosystem, competitive pricing, and a clear speed roadmap from 400GbE to 800GbE and beyond. With congestion management protocols like ECN and DCQCN, Ethernet fabrics deliver the lossless transmission that AI training demands.

East-west traffic patterns in AI training clusters

What is east-west traffic and why AI flipped the ratio

In traditional data centers, most traffic flows north-south: users request data from servers, servers respond back through the network perimeter. East-west traffic—server-to-server communication staying entirely within the data center—was secondary.

AI training clusters reverse this ratio.

“70 to 90 percent of all traffic in a GPU cluster is east-west: GPU talking to GPU, server talking to server, with almost nothing crossing the data center boundary during active training.”

The training data loads once (north-south). The gradient exchange that follows runs for days or weeks, producing continuous east-west traffic at line rate.

Where the traffic stays internal and where it hits the network

When GPUs train a model together, they split the work in several ways. Some of that work requires GPUs inside the same server to exchange data constantly—this is the heaviest and most latency-sensitive communication—and it stays entirely within the server on NVLink, a direct GPU-to-GPU interconnect running at 900 GB/s. No switches, no cables, no transceivers involved. The bulk of the cluster-wide coordination, synchronizing gradients between all GPUs through AllReduce operation, passing intermediate results between servers handling different stages of the model, and routing data to specialized components across the cluster flows out of the server and into the data center network. This is where optical transceivers enter the picture.

Each GPU connects to the network through its own dedicated network interface card at 400 Gbps today, moving to 800 Gbps in next-generation systems. An 8-GPU server pushes 3.2 Tbps of aggregate bandwidth into the fabric. Scale that to a 1,000-server cluster and the back-end network carries up to 3.2 Petabits per second of east-west traffic—all latency-sensitive, all lossless, and every byte passing through an optical transceiver on the way out and another on the way in.

Leaf-spine architecture optimization

Why standard data center network design falls short for AI

Modern data centers use a two-layer network design called leaf-spine. Picture it as two rows of switches. The bottom row (leaf switches) connects directly to servers; every server plugs into a leaf. The top row (spine switches) acts as a high-speed crossbar, where every bottom switch connects to every top switch. Any server reaching any other server always crosses exactly three hops (up to the top, across, and back down), there are multiple parallel paths to spread traffic across, and scaling is simple—add switches to either row as the cluster grows.

This design is already the standard for cloud and enterprise data centers. But a leaf-spine fabric built for general-purpose workloads will not survive AI training traffic. The synchronized bursts, elephant flows, and lossless requirements described in the previous sections demand a more aggressive build: lower oversubscription, higher-capacity spine switches, a different way of connecting GPUs to the bottom row, and uniform link speeds across every connection in the fabric.

Non-blocking design: why 1:1 matters

Oversubscription is the ratio between the bandwidth servers can push into the network versus the bandwidth available to carry it to the next layer. At 3:1 oversubscription, servers can collectively generate three times more traffic than the uplinks can handle.

In traditional data centers this works—not all servers send at full rate at the same time. In AI clusters, they do.

AllReduce forces every GPU to send simultaneously at line rate, and every GPU waits until the slowest one finishes. If the uplinks cannot carry the full load, buffers fill, packets drop, and the entire cluster slows down.

This is why AI back-end fabrics target 1:1 oversubscription, a non-blocking design where the uplink capacity matches the server capacity exactly. No traffic competing for insufficient bandwidth. The cost is real: more switches, more uplinks, more transceivers. But with a single 8-GPU server costing upwards of $400,000, spending more on the network to keep GPUs working is the better trade. Some validated AI fabric designs actually overprovision beyond 1:1 to ensure headroom during worst-case synchronized bursts. More switches and uplinks also means more transceivers; the difference between a 1:1 and 3:1 fabric can be thousands of additional optics.

Rail-optimized topology

In a standard design, all 8 GPUs in a server connect to the same switch. Rail-optimized topology spreads them out: each GPU connects to a different switch through its own dedicated NIC. GPU 0 in every server connects to Switch 0, GPU 1 to Switch 1, and so on, creating eight parallel “rails” across the cluster.

This means GPU 0 in Server A and GPU 0 in Server B share the same switch and are one hop apart. Training frameworks exploit this by scheduling the heaviest communication between GPUs on the same rail. Cross-rail traffic still takes a longer path through the spine, but it represents a smaller fraction of total demand.

The trade-off is cabling precision. Each GPU must connect to the correct switch; miswiring a single cable silently degrades performance for every GPU on that server. At thousands of servers, this demands rigorous labelling and validation testing before the cluster goes live.

Research from MIT and Meta shows that 99 percent of GPU pairs carry no traffic between them during LLM training, a finding that enables rail-only designs eliminating spine switches entirely and cutting transceiver counts by more than half. This also means 8 transceivers per server instead of 1–2 in traditional designs, multiplying the optics count at the access layer.

Rail-Optimized Topology:

• Optics only between leaf and servers
• Higher AI performance
• Lower latency between GPUs

Non-Rail Optimized Topology:

• Cable-optimized topology
• Lower AI performance
• 3x higher switch latency between GPUs

Rail-optimized design connects each GPU to a dedicated leaf switch, keeping traffic local. Classic fabric design routes all GPUs through the same leaf switch, requiring precise load balancing to avoid congestion.

What makes a transceiver AI-ready

Not every transceiver that works in a traditional data center will perform in an AI back-end fabric. The difference comes down to five factors that matter more at 400G+ speeds under AI workload conditions than they do in general-purpose networking.

PAM4 sensitivity

Every 400G and 800G transceiver uses PAM4 modulation, encoding two bits per symbol across four signal levels instead of the two levels used by older NRZ signaling.

The signal levels are closer together, making the link more sensitive to noise, temperature changes, and jitter. A transceiver operating within spec at 25°C may drift out of margin at 55°C case temperature in a GPU rack. At NRZ speeds, this drift was tolerable. At PAM4 speeds, it produces bit errors that FEC must correct, and when FEC cannot keep up, frames are dropped and RDMA retransmissions follow.

FEC overhead

Forward Error Correction is mandatory at 400G+ PAM4 speeds. KP4 FEC (IEEE 802.3, standard for 400GbE and 800GbE) adds approximately 50–100 nanoseconds of processing latency to every frame.

In an AI fabric where a 100-microsecond latency increase reduces training efficiency by 15 percent, FEC latency matters.

When a transceiver’s raw bit error rate sits close to the FEC correction threshold, the link operates in a gray zone. Frames pass FEC most of the time, but occasional bursts exceed correction capacity, producing CRC failures and packet drops. This intermittent behavior is the hardest failure mode to diagnose; the link looks healthy on average but creates tail latency spikes that drag down the entire AllReduce.

Where each type goes in the fabric

Different positions in the network have different requirements:

Server-to-leaf (under 5 meters, same rack): 400G-PDAC-QSFP-DD Direct Attach Copper (DAC) cables for lowest cost and power, or 400G-AOC-QSFP-DD Active Optical Cables (AOC) for slightly longer reaches. No pluggable transceiver needed; this saves power and cost at the highest-volume position.

Leaf-to-spine (5–100 meters, across rows): 400G-QSFP-DD-500 (400G-DR4) or 800G-OSFP800-500 (800G-DR8) pluggable transceivers over parallel single-mode fiber. This is the highest-volume pluggable position, and thermal reliability and consistent BER matter most here because these links carry aggregated traffic from entire racks.

Leaf-to-spine or spine-to-superspine (100m–2km, across buildings): 400G-QSFP-DD-2.1 (400G-FR4) or 800G-OSFP800-2.2 (800G-2xFR4) over duplex single-mode fiber. Longer reach means tighter optical budgets and higher sensitivity to connector contamination.

A smart procurement strategy does not use the same optic everywhere. Each position has a different thermal profile, power budget, and failure impact.

Telemetry capability

Modern transceivers support CMIS (Common Management Interface Specification) diagnostics: laser temperature, bias current, Tx/Rx optical power, supply voltage, and per-lane FEC error counters, all reported in real time. In an AI fabric with 10,000+ transceivers, this telemetry is how operations teams detect degradation before it becomes a link flap. A rising FEC correction trend on a specific transceiver predicts failure days in advance.

A transceiver with rich CMIS reporting is worth more than one with basic monitoring, because early detection prevents training job restarts that cost tens of thousands of GPU-hours.

The 400G to 800G transition

The industry is mid-transition. 400G is deployed at scale in AI clusters today; 800G spine switches are shipping.

The practical 2026 pattern: 400G at server access, 800G on spine uplinks. For the buyer, the question is whether current 400G purchases become stranded assets when 800G arrives at the access layer. The answer depends on whether the switches in the design support breakout modes (one 800G port split into two 400G). Planning the transceiver purchase alongside the switch lifecycle avoids wasting optics inventory when the fabric upgrades.

Link flaps and failure modes

GPU downtime costs $50 to $200+ per minute. A 1,000-GPU cluster with a 2 percent optics failure rate faces roughly 20 hours of downtime and $200,000 in losses—from a component representing 3 to 5 percent of total compute node cost.

A link flap, where a connection drops and re-establishes in rapid succession, can stem from thermal sensitivity, contaminated fiber connectors, or firmware issues. At scale, even occasional flaps are devastating: with 32,000 GPUs and one-hour checkpoint intervals, a single unstable link can waste 32,000 GPU-hours of compute. Operators report losing 20 to 30 percent of cluster productivity to persistent link instability before identifying and replacing the responsible transceivers.

Reducing GPU idle time from 33% to <15%

Bridging the gap

In unoptimized networks, GPU utilization sits in the 65 to 75 percent range, roughly a third of GPU capacity wasted waiting on the network. With proper optimization, utilization climbs above 90 percent. Architecture (non-blocking, rail-optimized, consistent speed) sets the ceiling. Closing the remaining gap requires operational optimization: smarter load balancing and higher link reliability.

Adaptive routing

Standard ECMP load balancing achieves only 50 to 60 percent effective bandwidth with AI elephant flows. Per-packet adaptive routing fixes this: the switch evaluates queue depth on every uplink for every packet and sends it down the least congested path, pushing effective bandwidth above 95 percent with up to 4.5x lower latency. Fixing load balancing is equivalent to doubling spine capacity without adding a single switch or transceiver.

Conclusion

The network connecting GPU servers is not passive infrastructure; it is an active participant in every training step. Every AllReduce synchronization, every gradient exchange, every intermediate result passed between pipeline stages flows through the data center fabric and through the optical transceivers that link it together.

Optical transceivers are central to all of it. They are the physical link through which every byte of GPU-to-GPU communication passes.

Signal quality determines GPU utilization. A transceiver with elevated bit error rate produces CRC failures, triggers RDMA retransmissions, and creates tail latency that holds back every GPU in the cluster. In a fabric where the slowest link sets the pace for thousands of GPUs, marginal optical performance is a measurable drag on training efficiency.

Link stability determines training uptime. A single link flap can halt a training job and waste tens of thousands of GPU-hours. Thermal drift, contaminated connectors, and firmware issues cause intermittent failures that are hardest to diagnose and most expensive in lost compute time.

Scale amplifies everything. A 1,000-GPU cluster requires over 10,000 transceivers in the back-end fabric alone. The difference between a 3.5W and 5W unit across 10,000 ports is 15 kilowatts. A 2 percent failure rate translates to $200,000 in GPU losses. A slightly higher bit error rate, invisible in a traditional data center, becomes a cluster-wide performance problem when multiplied across thousands of links that all feed into the same AllReduce GPU gradient exchange operation.

The transceiver is not the most expensive component in an AI training cluster. But it may be the one where the gap between “good enough” and “built for AI” has the largest impact on the return from every other component in the rack.