Sabina Anja, Chief Technologist and govt advisor at VMware Cloud Basis, talks with host Amey Ambade about how AI workloads reshape GPU community design, and why the community strikes into the crucial path of computation as soon as a mannequin is just too giant for a single GPU. Utilizing Llama 3’s 405 billion parameters as a place to begin, Sabina explains how sharding spreads mannequin weights throughout many GPUs. She then walks by the distinction between scaling up inside a server over NVLink and scaling out throughout servers over the cluster community. She describes collective site visitors as synchronous: no GPU strikes to the subsequent step till each GPU finishes, so one gradual GPU or congested hyperlink, referred to as a straggler, stalls the entire job. The dialog covers knowledge, tensor, and pipeline parallelism and the distinct site visitors sample every produces, with analogies corresponding to splitting a restaurant invoice to elucidate all-reduce. It then strikes to bodily topology, together with rail-optimized designs, devoted back-end AI networks that echo the storage space networks of the previous, and why oversubscription, a wise price wager in enterprise networks, fails when AI site visitors runs each hyperlink at full capability.
The dialogue then turns to congestion and latency: why synchronized collectives produce sudden bursts that overflow change buffers, how older strategies corresponding to precedence stream management return to pause site visitors earlier than loss, and the way overlapping communication with computation hides some community delay. On inference, Sabina explains how sharded fashions pay community latency on each generated token, why the prefill and decode phases have totally different useful resource calls for, and why some techniques separate them into totally different GPU swimming pools and transfer the KV cache between them. She additionally describes how mixture-of-experts fashions can focus site visitors on the servers that host standard consultants. The episode closes with a vendor-neutral framework for evaluating community materials: deal with the community as a design constraint in the identical means as compute and reminiscence, and provides the community group concrete details about the site visitors a workload will generate. Her abstract of the problem is that AI networking is a watermelon-through-a-keyhole downside.
Delivered to you by IEEE Pc Society and IEEE Software program journal.
Present Notes
Associated Episodes
SE Radio has robust protection of software and cluster networking however has not coated the specialised cloth that GPU workloads run on. These are the closest anchors; every connects at a selected level and differs in scope, which we will flag on air.

