AI infrastructure discussions often focus on accelerators, but large training and inference clusters are systems problems. A fast GPU can spend valuable time waiting for data or other GPUs if the network cannot keep up.

Distributed workloads communicate constantly

Training large models requires exchanging gradients and parameters across devices. Inference systems may also distribute model layers or route requests across accelerator pools. Communication overhead grows with scale. Bandwidth and latency solve different problems. High bandwidth moves large volumes of data, while low latency reduces the delay for frequent synchronization. Architecture needs both, depending on the communication pattern.

Topology matters.

Not every GPU-to-GPU path has the same cost. Placement decisions should account for which devices communicate most heavily. Poor topology can create hotspots even when aggregate network capacity appears sufficient. Congestion reduces expensive utilization. When multiple jobs compete for shared links, accelerators can stall. Monitoring should connect network congestion with GPU utilization and job performance rather than treating networking as a separate operations domain.

Storage is part of the path

Training data, checkpoints and model weights must move efficiently. Slow storage or oversubscribed links can delay job startup and recovery. In inference, model loading time also affects scaling behavior. Networking belongs in TCO. Accelerator acquisition is only one cost. High-performance networking, switches, optics, storage and operations contribute materially to cluster economics. Our AI infrastructure TCO framework shows why utilization and supporting systems need to be modeled together.

Design around workload shape.

A cluster optimized for large synchronous training jobs may not be ideal for diverse inference workloads. Capacity planning should begin with communication patterns, model sizes, service objectives and failure recovery needs. The practical lesson is simple: GPUs do not deliver performance in isolation. AI data centers are integrated compute, memory, network, storage, power and cooling systems, and bottlenecks move as workloads change. Separate interactive and elastic capacity. User-facing traffic benefits from warm, predictable resources while background jobs can often tolerate preemption or delay. Scheduling separation protects latency headroom without reserving every GPU permanently. Capacity can move between pools, but the policy should preserve SLOs before maximizing utilization.

Use queue signals before saturation

Queue age, pending tokens and admission delay can reveal overload earlier than GPU percentage. Control loops should combine several signals and use hysteresis to avoid oscillation. Model request heterogeneity. Prompt length, generation length and model size create very different memory and compute demand. Schedulers should account for request shape rather than treating every request as the same unit of work.

Define service objectives by traffic class.

Interactive inference and batch work have different goals. Real-time requests care about tail latency and time to first token, while offline jobs can trade latency for throughput. Explicit priorities prevent one workload from degrading another unpredictably. Capacity policy should state what may wait, what can route to an alternate model and what should be rejected during overload. Measure the whole request path. GPU utilization alone does not explain user latency. Queue time, model loading, KV-cache pressure, transfer and downstream processing can dominate. End-to-end tracing shows where the actual bottleneck lives.

Tail latency matters more than averages for interactive products.

Design overload behavior deliberately

During spikes, delay batch work, reduce optional processing, route to a smaller model or reject early before queues become unrecoverable. Bounded queues and admission control turn overload into a known operating mode. Separate interactive and elastic capacity. User-facing traffic benefits from warm, predictable resources while background jobs can often tolerate preemption or delay. Scheduling separation protects latency headroom without reserving every GPU permanently. Capacity can move between pools, but the policy should preserve SLOs before maximizing utilization.

Use queue signals before saturation.

Queue age, pending tokens and admission delay can reveal overload earlier than GPU percentage. Control loops should combine several signals and use hysteresis to avoid oscillation. Model request heterogeneity. Prompt length, generation length and model size create very different memory and compute demand. Schedulers should account for request shape rather than treating every request as the same unit of work.

Design overload behavior before an incident

When demand exceeds capacity, delay batch work, reduce optional processing, route eligible traffic to a smaller model or reject before queues become unstable. Controlled degradation protects the majority of users and shortens recovery.

Queue age and pending tokens often provide earlier warning than raw GPU utilization.

Connect performance with unit economics.

Warm pools, batching and model routing trade cost for latency or quality. Track cost per successful request by traffic class under realistic concurrency rather than optimizing utilization in isolation.