INSIGHTEngineering · Networking and fiber

Networking and fiber in an AI data center

An AI data center runs several networks at once, not one. A scale-up fabric binds accelerators inside a rack into a single compute domain, a scale-out backend fabric carries collective traffic between racks, and a separate frontend network handles storage, management, and the outside world. Fiber enters the building at one controlled point and fans out through a leaf-spine fabric — but almost all of the traffic that decides whether GPUs stay busy never leaves the building.

PUBLISHED 2026-08-31LAST VERIFIED 2026-08-31BY JOSEF ELIMELECHREVIEWED PODOS AI ENGINEERING

5
Traffic planes, one building: 800G
Ethernet defined by IEEE 802.3df-2024: 72
GPUs in one NVLink domain (NVL72)

What you need to know

Several networks, not one

Scale-up inside the rack, scale-out between racks, frontend and storage, out-of-band management, and external uplinks each carry different traffic and fail differently.

Fiber enters at one point

Outside-plant fiber terminates in an entrance room and cross-connects into the main distribution area. Two carrier paths in one conduit are one path with two invoices.

Training and inference are opposites

Training is synchronous bursts of elephant flows paced by the slowest path; inference is many small flows judged on tail latency. One fabric tuned for both is mediocre at each.

The failure is rarely a link going down

It is one optic degrading quietly and slowing every iteration of a job spanning thousands of GPUs — which only telemetry correlated with job metrics can find.

ENG-05Engineering

Four networks in one building

Enterprise data centers converge onto one general-purpose fabric. AI sites deliberately do not. Meta's published account of its training clusters describes separating GPU training onto its own dedicated backend network, because its traffic — bursty, load-imbalanced, coordinated across tens of thousands of GPUs for weeks at a time — is hostile to anything sharing a fabric with it.^[1]

Fiber entry and the physical plant

Everything external begins at the entrance facility: outside-plant fiber terminating in an entrance room, transitioning to inside-plant cable, cross-connecting into the main distribution area. ANSI/TIA-942 — the data-center telecommunications infrastructure standard, currently at revision C — defines those spaces, pathways, and redundancy topologies.^[5] Two carrier paths sharing one conduit are one path with two invoices; real diversity means separate entry points, conduits, and routes off the property. Inside, media follows the lane rate rather than taste. IEEE Std 802.3df-2024 defines 800 Gb/s Ethernet and 400/800 Gb/s physical layers on 100 Gb/s-per-lane signalling, with separate copper, multimode, and single-mode variants.^[4] Copper direct-attach survives a couple of metres at those lane rates, so it stays inside the rack; anything crossing a row runs on fiber, and single-mode dominates because its reach does not collapse as lane rates climb.

Leaf-spine: why the fabric looks the way it does

A leaf-spine fabric is a two-tier Clos topology: every leaf connects to every spine, and nothing connects leaf to leaf. Path length is uniform, so any server is the same number of hops from any other — which matters when a collective completes only as fast as its slowest participant. Load spreads across many equal-cost paths instead of one tree, and capacity grows by adding spines rather than replacing a chassis. The pressure point is oversubscription: the ratio of bandwidth facing servers to bandwidth facing spines. Backend AI fabrics are normally built non-blocking, because a bottleneck between racks is paid on every training step for the life of the cluster; frontend and storage planes are routinely oversubscribed, because their traffic tolerates queueing.

Hyperscale practice has already moved past the flat two-tier picture. Google's Jupiter work replaced a static spine layer with optical circuit switches under software-defined control, reporting 5× higher speed and capacity, a 30% capex reduction, and a 41% power reduction across the evolution period while serving live traffic.^[3] The transferable lesson for a smaller site is not optical switching — it is that topology has to be incrementally extensible, because the next hardware generation arrives before the building is full.

Reference

The planes, and what each one costs when it degrades

Five traffic planes share one building. They are separated because their failure modes are not interchangeable.

Plane	Scope	What it carries	If it degrades
NET-01	Scale-up fabric	All-to-all accelerator traffic inside one rack-scale domain.	The domain loses most of its throughput; jobs stall.
NET-02	Scale-out backend	Collectives between racks; pipeline stage handoffs.	Step time rises for every rank, not just one.
NET-03	Frontend and storage	Dataset reads, checkpoint writes, images, inference requests.	Data loaders starve GPUs; checkpoints land late.
NET-04	Out-of-band management	BMC and console access, firmware, sensor telemetry.	Operators lose remote hands when they most need them.
NET-05	External uplinks	Carrier transit, peering, remote operations, data ingress.	The site is unreachable while still computing.

A collective completes only as fast as its slowest participant — which is why a backend fabric is built non-blocking and every ratio above 1:1 is paid on every training step.
PODOS AI Engineering · backend fabric design

1:1

Target backend oversubscription

East-west traffic

Training and inference are different workloads

East-west means machine-to-machine traffic inside the facility. It dominates AI sites, but training and inference stress the fabric in opposite ways, and a fabric tuned for one is mediocre at the other.

Dimension	Distributed training	Inference serving
Traffic shape	Synchronous bursts — every rank transmits at the end of a step, then quiet.	Continuous and request-driven; smooth in aggregate, spiky per tenant.
Flow profile	Few very large elephant flows, deeply imbalanced across paths.	Many small flows, plus large KV-cache transfers when disaggregated.
Primary metric	Step time — the slowest path paces every rank.	Tail latency at a percentile, end to end.
Loss tolerance	Effectively zero; one retransmit stalls the collective.	Low but non-zero; a request retries without stalling the fleet.
Failure response	Restart from checkpoint — blast radius is the whole job.	Affected replicas drain; the service degrades, not stops.
Fabric requirement	Non-blocking backend, careful congestion control, tight failure domains.	Predictable queueing, spike headroom, graceful partial failure.

In operation

Scale-up boundaries, redundancy, and monitoring

Three things decide whether a fabric that looks right on a diagram keeps GPUs busy in production.

OPS-01

Where the scale-up boundary earns its keep

NVIDIA describes NVLink as a direct GPU-to-GPU interconnect that scales multi-GPU I/O within a server, with NVLink Switch chips extending all-to-all GPU communication across the rack^[6]; a GB200 NVL72 rack presents 72 GPUs and 36 CPUs as a single NVLink domain.^[7] Traffic between racks falls to Ethernet or InfiniBand instead. Parallelism that fits inside the scale-up domain never touches the fiber plant; parallelism that does not crosses a bandwidth cliff at the rack boundary, which makes scheduler placement a networking decision. The transport underneath is meanwhile consolidating on Ethernet: the Ultra Ethernet Consortium launched Specification 1.0 in June 2025, an RDMA transport for Ethernet and IP specified across NICs, switches, optics, and cables so multi-vendor fabrics interoperate without lock-in.^[2]

OPS-02

Redundancy is failure-domain arithmetic

Network redundancy in an AI site is failure-domain arithmetic. Dual-homing servers to two leaf switches, running N+1 spines, and feeding paired switches from separate power paths all reduce the number of accelerators one component can take out of a running job. That number is the design output: it sets how often jobs checkpoint, and therefore what a failure costs. The Uptime Institute's 2025 survey of more than 800 operators found roughly half had an impactful outage in the previous three years — redundancy is judged on what it contains, not on what is installed.^[8]

OPS-03

Monitoring has to reach past up and down

Monitoring has to reach past up and down: per-queue depth and drop counters, ECN marking and PFC pause rates on lossless fabrics, per-lane optical light levels, correctable-error counts, fabric-wide path utilisation. Alone these are graphs. They become diagnostic when correlated with job metrics — step time, collective completion time, rank stragglers — because the characteristic AI failure is not a link going down but one degrading optic quietly slowing every iteration of a job spanning thousands of GPUs.

Selecting a design

Network design checklist

The questions an engineering review actually asks, in roughly the order their answers constrain each other.

#	Criterion	What to evaluate	Design consequence
01	Scale-up domain size	Accelerators bound into one domain, and whether the model fits inside it.	Sets the rack boundary; parallelism that fits never touches the fabric.^[7]
02	Backend oversubscription	Leaf uplink capacity against accelerator port bandwidth beneath it.	Non-blocking is the default; any ratio above 1:1 is a per-step tax.
03	Transport and congestion control	RoCEv2 with PFC/ECN tuning, InfiniBand, or an Ultra Ethernet path.	Sets behaviour under incast — the most operationally sensitive choice here.^[2]
04	Media and lane rate	Copper reach in-rack versus optics between racks; breakout plans.	Fixes the optics count, which drives capex and component failure rate.^[4]
05	Cabling discipline	Entrance rooms, pathways, cross-connects, labelling against a standard.	Decides whether the plant is extensible in year five.^[5]
06	Failure domain sizing	Accelerators one leaf, optic, or feed removes from a running job.	Blast radius sets the checkpoint interval, and the cost of a failure.^[1]
07	Uplink diversity	Two carriers over physically separate entry paths and conduits.	Diversity on paper is not diversity in the ground.
08	Telemetry depth	Queue counters, ECN/PFC rates, optical light levels, job-step correlation.	Without correlation a degrading optic reads as a slow model.^[8]

HONEST LIMITS

When this architecture is not the right fit

A dedicated backend fabric is a commitment, and the cases below are where it does not pay for itself.

Single-rack and inference-only deployments do not need a dedicated backend fabric. If the work fits in one scale-up domain, a second non-blocking fabric buys operational surface, not throughput.
Lossless Ethernet is a discipline, not a purchase. Badly tuned PFC and ECN produce head-of-line blocking and congestion spreading — worse than a slower fabric nobody tuned.
Optics dominate the failure budget. A high-radix fabric holds more transceivers than any other active component, and they degrade gradually rather than fail cleanly.
Prefabricated units constrain topology. Cable lengths and cross-unit port counts are fixed at the factory, so multi-unit topologies are planned before manufacture, not patched in the field.
Standards are still moving. Ultra Ethernet 1.0 is recent; multi-vendor interoperability is something to test on real hardware rather than assume.

In the product

How PODOS treats the network as part of the unit

In a conventional build the cabling plant is field work: pathways pulled, fiber terminated, links tested on site after the shell is up. PODOS moves that work into the factory. Each PODOS Pod is designed as a standardized 1 MW building block and designed for 128 GPUs, so the internal fabric — racks, in-rack copper, intra-unit fiber, and the management plane — is a fixed, repeatable topology rather than a bespoke design per site. Site-side work reduces to landing external fiber on a defined interface at the unit boundary, which is part of why PODOS targets a 90-day window from order to commissioning for a standard unit.

Treating the unit as the failure domain lines the network up with the other physical domains: the power architecture feeding the racks and the cooling loop are scoped to the same boundary as the fabric inside it, so blast radius means one thing across all three. See the platform overview and deployment model for how units compose, use cases for the workloads these fabrics carry, modular vs traditional AI data centers for the comparison with a conventional facility, and the AI infrastructure glossary for unfamiliar terms.

QUESTIONS

Frequently asked questions

What is east-west traffic in an AI data center?

East-west traffic is machine-to-machine traffic inside the facility, as opposed to north-south traffic between the facility and the outside world. In AI clusters it dominates: training synchronises gradients across every participating GPU on every step, so the volume moving between racks dwarfs anything crossing the external uplink.

What is the difference between scale-up and scale-out networking?

Scale-up is the very high bandwidth fabric binding accelerators inside a node or rack into one domain — NVLink and its switch chips are the current example. Scale-out is the fabric between racks, on Ethernet or InfiniBand. Scale-up bandwidth per GPU is far higher, so schedulers keep tightly coupled parallelism inside one scale-up domain where they can.

Why is single-mode fiber used instead of copper inside AI clusters?

Copper direct-attach cables are cheaper and lower power, but at the lane rates current 400G and 800G optics run at, copper reach shrinks to a few metres, which confines it to intra-rack links. Anything crossing a rack or a row needs optics, and single-mode fiber carries those distances without the reach penalty multimode hits as lane rates climb.

How much oversubscription is acceptable in an AI backend fabric?

For the backend fabric carrying collectives the target is normally non-blocking, because a collective runs at the speed of its slowest path and oversubscription becomes a tax on every training step. Frontend, storage, and management networks are routinely oversubscribed, because their traffic tolerates queueing.

Sources

[1] RoCE networks for distributed AI training at scale (companion to the ACM SIGCOMM 2024 paper) — Meta Engineering, 5 Aug 2024
[2] Ultra Ethernet Specification 1.0 (announcement) — Ultra Ethernet Consortium, Jun 2025
[3] Jupiter Evolving: Transforming Google's Datacenter Network via Optical Circuit Switches and Software-Defined Networking (ACM SIGCOMM 2022) — Google Research, 2022
[4] IEEE Std 802.3df-2024 — Ethernet Amendment 9: 800 Gb/s MAC and 400/800 Gb/s physical layers — IEEE SA, approved Feb 2024
[5] ANSI/TIA-942 Telecommunications Infrastructure Standard for Data Centers (revision C) — Telecommunications Industry Association, revision C, accessed 2026-08-31
[6] NVLink and NVLink Switch product page — NVIDIA, accessed 2026-08-31
[7] GB200 NVL72 product page — NVIDIA, accessed 2026-08-31
[8] Global Data Center Survey 2025 (800+ operator respondents) — Uptime Institute, Jul 2025

Adjacent systems

ENGINEERING

Bring the fabric design to your site

Send the workload mix, the carrier options, and the site constraints. Engineering will tell you what the network inside a pod-based build looks like there.

Size your deployment See the deployment model