Ethernet

Ethernet is the family of wired computer networking technologies standardized by the IEEE 802.3 working group. It began in 1973 as a shared coaxial cable carrying about three megabits per second between office computers at Xerox PARC ^[1][2], and the faster generations that followed dropped the shared cable entirely for the switched, full duplex point-to-point links that fill a modern data center ^[9]. In AI infrastructure it has a specific job: Ethernet is the scale-out fabric, the network that binds thousands of accelerator servers into a single training cluster, and it is the main competitor to InfiniBand for that role.

Its position there is recent. Dell'Oro Group, which began tracking AI back-end networks in late 2023, found InfiniBand holding more than 80 percent share at the time ^[3]. By the third quarter of 2025 Ethernet accounted for more than two thirds of data center switch sales in AI back-end networks, up from less than half in the same period a year earlier, with 800 Gbit/s switches making up the bulk of shipments and revenue ^[4].

Reaching that point required changing a great deal about how Ethernet is built and operated. Training traffic looks nothing like web traffic: a handful of enormous, synchronized flows instead of millions of small independent ones, and a job whose progress is gated by the last message to arrive rather than by average throughput ^[5]. The adaptations include remote direct memory access over Ethernet (RoCEv2), lossless operation via priority flow control, and load balancing schemes that spread one flow across many paths. The Ultra Ethernet Consortium's 1.0 specification, released in June 2025, is the industry's attempt to replace the borrowed RDMA transport with one designed for this workload ^[6].

Origins at Xerox PARC

Robert Metcalfe circulated a memo at Xerox PARC in 1973 proposing a broadcast communication network to link the center's Alto computers, and the system he and David Boggs built ran at 2.94 Mbit/s ^[2]. Their 1976 paper in Communications of the ACM, "Ethernet: Distributed Packet Switching for Local Computer Networks," described an operating network of 100 nodes strung along a kilometer of coaxial cable, and set the design parameters of the experimental system at 1 kilometer, 3 megabits per second and 256 stations ^[1].

The paper explains the name plainly: "Ethernet is named for the historical luminiferous ether through which electromagnetic radiations were once alleged to propagate" ^[1]. The cable itself, called the Ether, was deliberately passive, because the authors chose "to distribute control of the communications facility among the communicating computers to eliminate the reliability problems of an active central controller" ^[1]. Stations listened for a carrier before transmitting, watched for interference while transmitting, briefly jammed the cable so that every colliding station would notice (a mechanism the paper calls collision consensus enforcement), and then retried after a random delay whose mean doubled with each successive collision, a heuristic the authors named binary exponential backoff ^[1]. Delivery was explicitly best effort: packets are delivered "only with high probability," and reliability was pushed up to higher-level protocols ^[1]. That decision, to keep the network simple and make endpoints responsible for correctness, helps explain how Ethernet later absorbed so many jobs it was never designed for.

Commercialization followed quickly. Metcalfe founded 3Com in 1979 ^[2]. In 1980 a consortium of Digital Equipment Corporation, Intel and Xerox published the DIX specification, a production quality 10 Mbit/s Ethernet ^[7]. The IEEE's LAN standards committee then took the technology up as 802.3, approving it on 24 June 1983 and issuing it as ANSI/IEEE Std 802.3-1985 ^[8][7]. ACM gave Metcalfe the 2022 A.M. Turing Award for his invention of Ethernet ^[2].

From shared cable to switched fabric

The Ethernet of the 1976 paper was a bus that every station shared. Through the 1980s and 1990s the medium shifted to twisted pair and the topology became a star, centered first on repeater hubs and then on switches. Once each station had its own switch port running full duplex, collisions stopped happening and the carrier sense machinery that defined early Ethernet became vestigial. It was dropped outright at higher speeds: 10 Gigabit Ethernet "defines only full-duplex point-to-point links," and shared-medium CSMA/CD operation "has not been carried over from the previous generations of Ethernet standards" ^[9].

What survived is the frame format, the addressing model, and the ecosystem. The Ultra Ethernet Consortium's own founding rationale lists that ecosystem as the reason AI clusters gravitate to Ethernet at all: a broad multi-vendor supply of switches, NICs, cables, transceivers and management software; the proven addressing and routing scale of IP networks; mature tooling for testing and operations; and a competitive market that drives cost down ^[5].

Scroll sideways for more →

Why AI clusters stress the network

Distributed training splits a model across many accelerators using data parallelism, model parallelism or both, and alternates compute phases with communication phases. The next phase cannot start until the collective operation finishes, so the last message to arrive gates the progress of every participating GPU, which makes tail latency rather than average bandwidth the metric that matters ^[5]. All-reduce and all-to-all are the dominant collective types in these workloads ^[5].

Three properties of that traffic work against a network designed for the internet. First, there are very few flows and each is enormous. Equal-cost multipath routing hashes a flow onto a single path, which confines a high throughput flow to one link and degrades performance when too many flows land on the same path ^[5]. Meta reported that standard ECMP performed poorly on its training fabric precisely because of the low flow entropy of the workload ^[20]. Second, all-to-all patterns create incast, where many senders converge on one receiver's final link at the same moment ^[5]. Third, the RDMA transports inherited from InfiniBand recover from a single lost or reordered packet with Go-Back-N retransmission, resending packets the receiver already has and lowering goodput ^[5].

RDMA over Converged Ethernet

The bridge between Ethernet and AI clusters is RoCE, and in practice RoCEv2. As the SIGCOMM 2015 paper that introduced DCQCN describes it, RoCEv2 "retains the IB transport layer, but replaces IB networking layer (L3) with IP and UDP encapsulation, and replaces IB L2 with Ethernet. The IP header is needed for routing, while the UDP header is needed for ECMP" ^[21]. Meta describes the same encapsulation in operational terms: RoCEv2 wraps the RDMA service in UDP packets so it can traverse a routed network, giving lossless, high bandwidth, low latency transport between any two GPUs ^[20].

To run efficiently, RoCE has historically required a lossless Layer 2. That is supplied by IEEE 802.1Qbb priority-based flow control, approved in June 2011, which enables flow control per traffic class on full duplex links so that loss-sensitive protocols can share a link with ordinary LAN traffic ^[22]. PFC works, but it is coarse. It operates per port, or per port and priority, and does not distinguish between flows, which causes head-of-line blocking and congestion spreading ^[21]. DCQCN, developed by Microsoft, Mellanox and UC Santa Barbara, added an end-to-end congestion control scheme for RoCEv2 that needs only standard RED and ECN support in switches, with the rest implemented on host NICs ^[21].

The reason operators tolerate this complexity is the performance gap. In the DCQCN paper's testbed, average user-level latency for a 2 KB transfer was 25.4 microseconds over TCP versus 1.7 microseconds for RDMA read and write and 2.8 microseconds for send, and driving full throughput with 4 MB messages consumed on average more than 20 percent of CPU cycles across all cores for TCP while the RDMA client used under 3 percent ^[21].

Operational experience since then has been mixed and instructive. Meta tried to tune DCQCN for its 400G deployments, found that a change to the firmware implementation had introduced bugs, and proceeded without it. Writing in August 2024, it reported over a year of running on PFC alone with no other transport-level congestion control, observing stable performance and no persistent congestion for training collectives, and said it had not seen a case in four years of production AI training traffic forcing a cluster training switch to send flow control pauses to rack switches persistently. Part of the credit goes to receiver-driven traffic admission, in which the sender posts an RDMA write only after a clear-to-send packet arrives from the receiver, with those clear-to-send packets given high priority queuing at the switches ^[20]. The Ultra Ethernet Consortium's white paper takes the opposite lesson, arguing that large lossless RoCE networks "require careful tuning, operation, and monitoring to perform well without triggering these effects," that this expertise is not available to every operator, and that a transport which does not depend on a lossless fabric is therefore needed ^[5].

Load balancing and adaptive routing

Meta's published progression through load balancing schemes is a compact history of the problem. Static ECMP was inadequate. Path pinning worked until a rack was only partially allocated to a job, at which point uneven traffic distribution degraded performance by more than 30 percent. The fix was enhanced ECMP, in which switches hash on the destination queue pair field so that a single connection's packets spread across paths, which improved the all-reduce collective by up to 40 percent ^[20]. Meta's clusters are built as a two-stage Clos topology called an AI Zone, with rack training switches connecting the GPUs inside a rack over copper cables, cluster training switches providing scale-out connectivity among all racks over single-mode fiber and 400G transceivers, and an aggregator layer that ties the cluster switches together within a data center building ^[20].

NVIDIA Spectrum-X attacks the same problem from the vendor side, pairing Spectrum-4 Ethernet switches with BlueField-3 SuperNICs. Instead of static hashing it performs fine-grained adaptive routing, load balancing "packet-by-packet across the network, without the need for deep buffers," while the SuperNIC reorders packets at the destination so the reordering stays invisible to the application ^[23]. Its congestion control is telemetry driven, combining high-frequency telemetry probes with flow metering to isolate the performance of tenants sharing one fabric ^[23]. NVIDIA claims the platform accelerates AI network performance by 1.6 times over off-the-shelf Ethernet ^[24]. Its largest publicly documented deployment is xAI's Colossus, a cluster of 100,000 NVIDIA Hopper GPUs in Memphis, Tennessee, which NVIDIA said in October 2024 used Spectrum-X for its RDMA network and sustained 95 percent data throughput with no packet loss from flow collisions ^[25]. In August 2025 NVIDIA added Spectrum-XGS, a scale-across variant for joining distributed data centers, which it says nearly doubles NCCL performance across separated clusters using distance-adjusted congestion control, latency management and end-to-end telemetry ^[26].

Ethernet versus InfiniBand

The clearest public comparison came from Meta, which in March 2024 described two clusters of 24,576 NVIDIA H100 GPUs each, built to evaluate both options directly. One used a RoCE fabric based on the Arista 7800 with Wedge400 and Minipack2 OCP rack switches; the other used an NVIDIA Quantum2 InfiniBand fabric. Both interconnect 400 Gbit/s endpoints ^[27]. Meta reported using both fabrics for large generative AI workloads, including the training of Llama 3 on the RoCE cluster, "without any network bottlenecks" ^[27].

That result, plus cost and supply considerations, is roughly what the market followed. Dell'Oro's July 2025 assessment was that Ethernet was "firmly positioned to overtake InfiniBand in these high-performance deployments," with nearly 80 billion dollars in data center switch sales at stake over five years ^[3]. InfiniBand has not disappeared; Dell'Oro attributed a surge in InfiniBand switch sales in the third quarter of 2025, its second consecutive quarter of strong growth, to the rapid ramp of NVIDIA's Blackwell Ultra platform, even as Ethernet held the overall lead ^[4].

The Ultra Ethernet Consortium

The Ultra Ethernet Consortium was announced on 19 July 2023 as an open Linux Foundation Joint Development Foundation project, with AMD, Arista, Broadcom, Cisco, Eviden (an Atos business), HPE, Intel, Meta and Microsoft as founding members ^[28]. Its charter covers protocols and signaling characteristics for Ethernet, "link-level and end-to-end network transport protocols to extend or replace existing link and transport protocols," congestion and telemetry mechanisms suited to AI, ML and HPC environments, and the surrounding software, storage, management and security constructs, organized into physical, link, transport and software layer working groups ^[28].

UEC published Specification 1.0 on 11 June 2025, covering all layers of the stack including NICs, switches, optics and cables, with named work areas in modern RDMA for Ethernet and IP, transport and congestion control, direct memory access, link and PHY technologies, and network security ^[6]. The consortium operates under the Linux Foundation Joint Development Foundation; its steering members include AMD, Arista, Broadcom, Cisco, HPE, Intel, Meta, Microsoft and Oracle, and maintenance updates have carried the specification to version 1.0.3 ^[29].

The design targets set out in UEC's founding white paper give a good picture of what Ultra Ethernet Transport is meant to do: an open protocol running over IP and Ethernet; multipath packet spraying that uses the whole fabric without causing head-of-line blocking, removing the need for centralized load balancing controllers; incast management that limits fan-in on the destination link with minimal drop; rate control that ramps quickly to wire rate without penalizing competing flows; out-of-order packet delivery with optional in-order message completion; support for 1,000,000 endpoints; good utilization without workload-specific congestion tuning; and wire-rate operation on commodity hardware at 800G, 1.6T and beyond ^[5]. The transport also encrypts and authenticates traffic between endpoints in a training or inference job by design ^[5].

Below the transport, Ultra Ethernet adds link layer mechanisms: link level retry, a hop-by-hop scheme in which packets are buffered at the sender until the receiver acknowledges them, and credit-based flow control, in which senders transmit only when the receiver confirms available buffer space, which reduces buffering requirements compared with PFC ^[30]. Congestion signaling uses a compare-and-replace model so each device along a path can overwrite the packet with more severe congestion information, giving the sender a picture of which links are congested within one round trip ^[30]. Technical summaries of the released specification describe three profiles: AI Base for bandwidth-bound collectives in training and inference, AI Full as a superset adding features such as tagging and rendezvous support, and HPC with MPI semantics and hardware tag matching ^[31].

Scale-up versus scale-out

A recurring confusion in AI networking is treating every accelerator interconnect as a network of the same kind. Scale-up fabrics provide high bandwidth and low latency between nearby accelerators; current implementations are vendor proprietary and subject to proximity limits, typically confined to a rack or a row ^[32]. Scale-out networking is the layer that connects those racks to each other across the cluster ^[20]. Ethernet, in the form of RoCE and now Ultra Ethernet, is a scale-out technology.

NVLink is not Ethernet. It is NVIDIA's proprietary interconnect with its own physical layer, protocol and switch silicon (NVSwitch). NVIDIA states that the fifth generation delivers 1.8 TB/s per GPU with 130 TB/s of aggregate bandwidth in an NVL72 rack, and that the sixth generation reaches 3.6 TB/s per GPU and 260 TB/s aggregate for a 72-GPU Vera Rubin NVL72 in an all-to-all topology, which it puts at more than 14 times the bandwidth of PCIe Gen6 ^[33]. Those figures are an order of magnitude above what any Ethernet port delivers, which is why the two coexist rather than compete directly.

The industry is nonetheless trying to move scale-up onto Ethernet framing. Broadcom began shipping Tomahawk Ultra on 15 July 2025: a 51.2 Tbit/s switch that holds 250 nanosecond switch latency at full throughput, cuts Ethernet header overhead from 46 bytes to as low as 10 while keeping full Ethernet compliance, and implements link level retry and credit-based flow control for a lossless fabric ^[34][35]. In October 2025 the Open Compute Project launched a workstream called Ethernet for Scale-Up Networking (ESUN) with AMD, Arista, Arm, Broadcom, Cisco, HPE Networking, Marvell, Meta, Microsoft, NVIDIA, OpenAI and Oracle. ESUN restricts itself to "open, standards-based Ethernet switching and framing for scale-up networking," starting with L2 and L3 framing and switching for lossless, error-resilient single-hop and multi-hop topologies, and explicitly excludes host-side stacks, non-Ethernet protocols and proprietary technologies ^[32].

Silicon, systems and hyperscaler transports

Switch silicon capacity has climbed sharply over the same period. Broadcom's Tomahawk 5, shipped from August 2022, provides 51.2 Tbit/s of Ethernet switching capacity in a single monolithic device ^[36]; Tomahawk 6, which began shipping on 3 June 2025, delivers what Broadcom called the world's first 102.4 Tbit/s of switching capacity in a single chip, double the bandwidth of any Ethernet switch then on the market, aimed at both scale-up and scale-out AI networks and offered with support for co-packaged optics ^[37]. Those chips end up in systems: Meta built 51T fabric switches on Broadcom and Cisco silicon for its Disaggregated Scheduled Fabric, which presents "an open and standard Ethernet-based RoCE interface to endpoints and accelerators" across GPUs and NICs from several vendors and runs on the OCP-SAI standard with Meta's FBOSS network operating system ^[38]. Optics is the other frontier; NVIDIA claims its Spectrum-X Ethernet Photonics switches, which package the optics with the switch itself, deliver five times better network power efficiency than traditional pluggable transceiver-based networks ^[24].

Not every large operator waited for a consortium. Google announced Falcon at the OCP Global Summit on 18 October 2023, a hardware-assisted Ethernet transport that supports RDMA and NVMe upper layer protocols, runs over standard Ethernet and IP networks, and was already implemented in Intel's IPU E2000 series; Google committed to contributing the specification to the Open Compute Project ^[39]. The pattern behind Falcon, Ultra Ethernet Transport and NVIDIA's RoCE extensions is the same one: keep Ethernet as the wire and the routing layer, and replace the transport above it.

Limitations and open questions

Lossless Ethernet remains the awkward part of the design. PFC prevents drops but spreads congestion and blocks unrelated traffic behind a paused queue, and DCQCN's sensitivity to the latency, speed and buffering of the underlying network means its tuning does not transfer between fabrics ^[21][5]. Packet spraying solves the load balancing problem but pushes reordering onto the endpoint, which means it works only with NICs built for it ^[23][5].

The standardization picture is also unsettled. Ultra Ethernet Transport depends on endpoints that can spray packets across every path and reassemble them out of order, which means new NIC silicon rather than a software upgrade ^[5][23], while deployed AI back-end fabrics are still overwhelmingly built on 800 Gbit/s Ethernet switches ^[4]. The 1.6 Tbit/s generation is still a draft: P802.3dj remains an active IEEE 802.3 task force project rather than an approved standard ^[18][19]. And scale-up has at least three competing answers, with NVLink and NVLink Fusion on one side, UALink on another, and ESUN pulling toward Ethernet framing ^[32]. Which of them ends up inside the rack will determine how much of the AI data center Ethernet eventually covers.

See also

• InfiniBand
• Ultra Ethernet
• NVIDIA Spectrum-X
• Broadcom Tomahawk 6
• NVLink
• GPU cluster

References

1. ^Robert M. Metcalfe and David R. Boggs, "Ethernet: Distributed Packet Switching for Local Computer Networks," Communications of the ACM, Volume 19, Number 7, July 1976, pp. 395-404. cl.cam.ac.uk/...metcalfe1976ethernet.pdf
2. ^MIT News, "Bob Metcalfe '69 wins $1 million Turing Award," 22 March 2023. news.mit.edu/...etcalfe-wins-acm-turing-award-0322
3. ^Dell'Oro Group, "Ethernet is Winning the War Against InfiniBand in AI Back-End Networks," 15 July 2025. delloro.com/...-infiniband-in-ai-back-end-networks
4. ^Dell'Oro Group, "AI Back-End Networks Continue Their Shift to Ethernet," 5 December 2025. delloro.com/...ks-continue-their-shift-to-ethernet
5. ^Ultra Ethernet Consortium, "Overview of and Motivation for the Forthcoming Ultra Ethernet Consortium Specification," 2023. ultraethernet.org/...-Overview-FINAL-WITH-LOGO.pdf
6. ^Ultra Ethernet Consortium, "UEC Launches Specification 1.0, Transforming Ethernet for AI and HPC at Scale," 11 June 2025. ultraethernet.org/...ernet-for-ai-and-hpc-at-scale
7. ^Charles Spurgeon, "Ethernet Quick Reference: A Brief History of Ethernet." ethermanage.com/...ch1qr_4
8. ^Engineering and Technology History Wiki (IEEE History Center), "Milestones: Ethernet Local Area Network (LAN), 1973-1985." ethw.org/...et_Local_Area_Network_(LAN),_1973-1985
9. ^Wikipedia, "10 Gigabit Ethernet." en.wikipedia.org/...10_Gigabit_Ethernet
10. ^Wikipedia, "Fast Ethernet" (IEEE 802.3u-1995, 26 October 1995). en.wikipedia.org/...Fast_Ethernet
11. ^IEEE 802.3z Gigabit Task Force. ieee802.org
12. ^IEEE 802.3ab 1000BASE-T Task Force. ieee802.org/...ab
13. ^IEEE P802.3ae 10Gb/s Ethernet Task Force. ieee802.org/...ae
14. ^IEEE P802.3ba 40Gb/s and 100Gb/s Ethernet Task Force. ieee802.org/...ba
15. ^IEEE P802.3by 25 Gb/s Ethernet Task Force. ieee802.org/...by
16. ^IEEE P802.3bs 200 Gb/s and 400 Gb/s Ethernet Task Force. ieee802.org/...bs
17. ^IEEE P802.3df 400 Gb/s and 800 Gb/s Ethernet Task Force. ieee802.org/...df
18. ^IEEE P802.3dj 200 Gb/s, 400 Gb/s, 800 Gb/s, and 1.6 Tb/s Ethernet Task Force. ieee802.org/...dj
19. ^IEEE 802.3 Ethernet Working Group. ieee802.org
20. ^Meta Engineering, "RoCE networks for distributed AI training at scale," 5 August 2024. engineering.fb.com/...ributed-ai-training-at-scale
21. ^Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia and Ming Zhang, "Congestion Control for Large-Scale RDMA Deployments," ACM SIGCOMM 2015, London, pp. 523-536. conferences.sigcomm.org/...p523.pdf
22. ^IEEE 802.1Qbb, Priority-based Flow Control (approved 16 June 2011). 1.ieee802.org/...802-1qbb
23. ^NVIDIA Developer Blog, "Optimize Large-Scale AI Workloads with NVIDIA Spectrum-X." developer.nvidia.com/...ads-with-nvidia-spectrum-x
24. ^NVIDIA, "Spectrum-X Ethernet Platform for AI." nvidia.com/...spectrumx
25. ^NVIDIA, "NVIDIA Ethernet Networking Accelerates World's Largest AI Supercomputer, Built by xAI," 28 October 2024. nvidianews.nvidia.com/...t-networking-xai-colossus
26. ^NVIDIA, "NVIDIA Introduces Spectrum-XGS Ethernet to Connect Distributed Data Centers Into Giga-Scale AI Super-Factories," 22 August 2025. investor.nvidia.com/...default
27. ^Meta Engineering, "Building Meta's GenAI Infrastructure," 12 March 2024. engineering.fb.com/...g-metas-genai-infrastructure
28. ^Joint Development Foundation, "Leading Cloud Service, Semiconductor, and System Providers Unite to Form Ultra Ethernet Consortium," 19 July 2023. jointdevelopment.org/...-ultra-ethernet-consortium
29. ^Ultra Ethernet Consortium home page (steering members and specification downloads). ultraethernet.org
30. ^Cisco Blogs, "Ultra Ethernet for Scalable AI Network Deployment." blogs.cisco.com/...-scalable-ai-network-deployment
31. ^Glenn K. Lockwood, "Ultra Ethernet" technical notes. glennklockwood.com/...ultra-ethernet
32. ^Network World, "Major network vendors team to advance Ethernet for scale-up AI networking," 14 October 2025. networkworld.com/...net-for-scale-up-ai-networking
33. ^NVIDIA, "NVLink and NVLink Switch." nvidia.com/...nvlink
34. ^Broadcom, "Broadcom Ships Tomahawk Ultra: Reimagining the Ethernet Switch for HPC and AI Scale-up," product news release, 15 July 2025. broadcom.com/...63341
35. ^Evertiq, "Broadcom ships Ethernet switch Tomahawk Ultra for HPC and AI scale-up," 24 July 2025. evertiq.com/...ahawk-ultra-for-hpc-and-ai-scale-up
36. ^Broadcom, "Broadcom Ships Tomahawk 5, Industry's Highest Bandwidth Switch Chip to Accelerate AI/ML Workloads," 16 August 2022. broadcom.com/...60456
37. ^Broadcom, "Broadcom Ships Tomahawk 6: World's First 102.4 Tbps Switch," 3 June 2025. broadcom.com/...63146
38. ^Meta Engineering, "Meta's open AI hardware vision," 15 October 2024. engineering.fb.com/...metas-open-ai-hardware-vision
39. ^Google Cloud Blog, "Introducing Falcon: a reliable low-latency hardware transport," 18 October 2023. cloud.google.com/...low-latency-hardware-transport