What began as discrete AI model training and human-facing chat interfaces has evolved into always-on AI factories dedicated to producing intelligence at scale. These factories are now tasked with powering agentic workflows that reason, plan, use tools, verify intermediate results, and execute complex multistep tasks across vast contexts.

Agentic workloads are not defined by a single prompt and response, but by sustained inference across many reasoning steps. They demand low per-step latency, high decode throughput, efficient long-context attention, large KV cache capacity, and the ability to scale models across tightly coupled GPU domains. The data center must be reimagined as a single unit of compute, a vision realized with the NVIDIA Vera Rubin platform.

At the core of the platform is the NVIDIA Rubin GPU, designed to deliver up to 10x more agentic throughput per unit of energy than NVIDIA Blackwell (Figure 1). Enhanced Tensor Cores with expanded precision flexibility, a new HBM4 memory subsystem, and the third-generation Transformer Engine—delivering up to 50 petaflops of NVFP4 performance—work together to accelerate agentic workloads efficiently.

Chart comparing agent throughput and interactivity across NVIDIA Hopper, Blackwell, and Rubin systems, showing Rubin NVL72 plus Vera supports roughly 10 times more agents and twice as many tool calls.
Figure 1. Pareto frontiers illustrating the 10x generational uplift in agentic inference performance of the Vera Rubin platform (internal 2T MoE workload)

This post examines how the NVIDIA Rubin GPU and its co-designed scale-up system address the end-to-end bottlenecks of agentic inference, from data movement and compute efficiency to long-context execution and rack-scale deployment. 

How does Rubin GPU architecture support agentic workloads?

The Rubin GPU (Figure 2) is constructed from reticle limited compute dies to achieve high density and efficiency. These two dies are unified on a single package through a high-speed inter-die link called the NVIDIA High-Bandwidth Interface (NV-HBI).

Annotated diagram of the NVIDIA Rubin GPU, showing graphics processor clusters, HBM memory controllers, L2 cache, NVLink, PCIe Gen 6, and performance specifications.
Figure 2. NVIDIA Rubin GPU chip architecture

The architecture begins with the challenge of keeping an enormous amount of compute productive as agentic workloads shift between reasoning, generation, retrieval, and tool use. Its 336 billion transistors, 224 streaming multiprocessors (SMs), and 896 Tensor Cores provide the raw compute density, while the third-generation Transformer Engine adapts precision across numerical formats. That flexibility enables the Rubin GPU to deliver up to 50 petaflops of NVFP4 inference performance while preserving accuracy.

But performance depends on more than Tensor Core throughput. The Rubin GPU organizes compute resources into Graphics Processor Clusters (GPCs) with a large centralized L2 cache. The GigaThread Engine coordinates work, MIG Control partitions the GPU for multiple workloads, and NV-DEC accelerates decoding. Together, these capabilities help the Rubin GPU turn compute density into sustained utilization across the diverse and dynamic workloads that define large-scale agentic systems.

That utilization also depends on how quickly data can reach compute cores. Rubin integrates up to 288 GB of HBM4 memory, driven by dedicated HBM controllers and 12-Hi stacks, to deliver up to 22 TB/s of peak bandwidth. The enhanced Tensor Memory Accelerator (TMA) manages high-efficiency movement across complex data layouts, while NVIDIA NVLink 6 provides 3,600 GB/s scale-up bandwidth to the NVLink Switch for all-to-all GPU-GPU communication, NVLink-C2C delivers 1,800 GB/s for coherent CPU-GPU communication, and x16 PCIe Gen 6 provides up to 256 GB/s of host connectivity.

Finally, large-scale agentic deployments must protect data as it moves through this execution domain. Confidential Computing with TEE-I/O is designed to secure data at rest, in transit, and in use across the AI factory. Together, these compute, memory, connectivity, and security capabilities form the GPU-level foundation for agentic workloads that must sustain efficient execution across increasingly large models and scale-up domains.

How does the Rubin GPU accelerate critical inference paths? 

Peak compute alone is not enough to accelerate agentic inference. Real-world performance also depends on how efficiently the GPU moves data, executes matrix operations, processes long-context attention, and transitions between dependent kernels. This section explores the Rubin GPU features designed to reduce overhead in these critical execution paths. 

Accelerating rack-scale MoE weight and token movement

Mixture-of-experts (MoE) models dynamically route tokens across many expert networks. As the number of experts grows, efficiently locating and moving expert weights becomes increasingly important to inference performance.

The Rubin GPU enhances the Tensor Memory Accelerator to reduce data-movement overhead for expert-heavy models. Its improved descriptor handling allows software to work more efficiently with tensors that share common layouts but reside at different locations in memory.

Rubin improves this with inline descriptor update support for TMA (Figure 3). Instead of modifying the descriptor in memory, the kernel can keep one unified descriptor for tensors that share the same layout and override fields such as the memory pointer and stride directly in the TMA instruction at runtime. 

Side-by-side Blackwell and Rubin diagrams: Blackwell uses one MoE descriptor per expert, while Rubin shares one descriptor across all experts.
Figure 3. Rubin simplifies MoE descriptor sharing

This helps MoE models scale more efficiently as expert counts increase. By reducing metadata-management and data-movement overhead, Rubin enables more GPU time to be devoted to useful inference computation, supporting higher throughput for agentic workloads that rely on large MoE models.

Doubling the efficiency of rack-scale matrix operations

Rubin doubles the Tensor Core throughput per clock by doubling the amount of data it can process along the \(K\) dimension. This optimization not only helps throughput bound kernels but also memory and latency bound kernels.

As model execution is split across many GPUs, each GPU often receives a smaller slice of the output work while the reduction dimension remains large. 

The benefit of a larger \(K\) dimension is fewer \(K\) loop iterations. In Figure 4, a GEMM that requires four \(K\) iterations on Blackwell can be completed with two iterations on Rubin. Fewer iterations reduce loop overhead, improve Tensor Core utilization, and help both context and decode GEMMs run more efficiently at high tensor-parallel scale.

Diagram showing Rubin processes a two-times larger K-dimension in two iterations, compared with four iterations on Blackwell.
Figure 4. Rubin doubles K-dimension instruction throughput

Addressing key challenges in long-context processing for agentic AI

Long-context and agentic AI workloads put increasing pressure on attention. As context windows grow, the model must compare more tokens, normalize larger attention-score matrices, and apply those scores to the value data used to produce the next layer’s output. This makes attention one of the most important performance paths for improving tokens per second per user.

Rubin accelerates attention by combining activation sparsity with adaptive compression and improved softmax throughput. One simple, safe, and effective way to use Rubin’s new sparsity features in attention is as follows. The attention pipeline begins with a dense \(QK^T\) computation to generate the intermediate attention scores. Rubin can then load that intermediate data from Tensor Memory into a structured 2:4 sparse compressed form, generating both nonzero values and the metadata needed to use them efficiently while reducing the scores’ write cost and storage requirements. This allows the later attention stages to operate on less data while preserving the dense output format expected by the rest of the model.

Diagram showing a dense matrix converted to a sparse matrix with metadata, applied to attention and MLP activation stages in a transformer block.
Figure 5. Rubin sparsity features can be applied to the attention block and MLP block activations

This compressed intermediate representation reduces work in two key places: softmax and the second attention GEMM. Softmax can operate on the nonzero attention values, and the following multiplication with the dense \(V\) matrix can use sparse MMA with the nonzeros from Softmax and the metadata from the original compression step. The result is less compute and data movement in the most expensive part of long-context attention improving tokens/watt, without requiring the surrounding model pipeline to change its interface.

Rubin also improves the softmax throughput. As Tensor Core throughput increases, softmax can become a bottleneck because it depends on exponential math and reductions across attention rows. Rubin increases exponential throughput, including 2x FP32 and 4x BF16/FP16 throughput versus the Blackwell baseline, helping softmax keep pace with faster matrix operations.

NVIDIA GPU platformFP32 exponential throughput
(per clk per SM )		BF16/FP16 exponential throughput
(per clk per SM)
Blackwell1x1x
Blackwell Ultra2x2x
Rubin2x4x
Table 1. Exponential throughput rises from Blackwell to Rubin

 Together, these features make the Rubin attention acceleration more than a single-kernel improvement. Activation sparsity reduces the amount of intermediate attention work, while faster exponentials reduce the softmax bottlenecks. 

Improving kernel execution efficiency

As inference scales across larger models and more GPUs, raw Tensor Core throughput is only part of the performance story. The GPU also needs to move efficiently from one kernel to the next. In inference, this is especially important because activations often sit on the critical path: one kernel produces activation data, writes it to memory, and the next kernel consumes that data to continue generating the next token.

Traditional producer-consumer execution can create bubbles in the GPU timeline. A producer kernel may complete work for some tiles or thread blocks early, but the consumer may not begin useful work until a broader dependency is resolved. The Blackwell programmatic dependent launch improves this by allowing earlier consumer-kernel progress, but dependent work can still wait for required activation data to become available.

Blackwell and Rubin timelines compare producer and consumer thread blocks; Rubin uses data-driven polling to begin consumer work sooner as producer data becomes available.
Figure 6. Producer-consumer overlap: Blackwell bulk triggering (above) and Rubin tile-level triggering (below)

Rubin enables more fine-grained coordination between dependent kernels. This allows consumer work to begin earlier as required input data becomes available, rather than waiting for a larger set of producer work to complete.

The result is a more tightly packed GPU timeline, reduced idle gaps, and improved overlap between dependent kernels. This is particularly valuable for agentic inference, where activations move sequentially through the model and kernel-to-kernel latency directly affects tokens per second per user.

How do Rubin memory and communication sustain high-throughput inference? 

As models, context windows, and GPU domains grow, data movement becomes as important as computation. Rubin is designed to improve the flow of weights, activations, KV cache data, and communication traffic within the GPU and across scale-up systems. The following sections look at the memory and communication innovations that help sustain high-throughput inference.

Accelerated scale-up communications

As inference scales from a single GPU to full rack-level systems, communication becomes part of the critical performance path. When communication is fused directly inside a GPU kernel, the kernel does not stop and hand control back to the CPU; it directly writes data or performs reductions over NVLink to another GPU while computation is still in flight.

Traditional GPU-to-GPU communication requires coordination and synchronization work in addition to moving payload data. These steps can add latency and consume interconnect bandwidth, particularly when communication occurs frequently within distributed inference workloads.

Rubin introduces counted writes for device-initiated NVLink communication. This capability streamlines synchronization for GPU-to-GPU data transfers by allowing the receiving GPU to track transfer completion more efficiently.

Side-by-side communication diagrams show Blackwell using memory barriers, acknowledgments, and atomic flags, while Rubin uses a counter update before the receiving GPU loads data.
Figure 7. Rubin accelerates NVLink communications using counted writes

NVLink fused communication with counted writes provides a lower-latency mechanism for coordinating GPU-to-GPU data movement, helping keep computation moving instead of waiting on synchronization operations. 

Co-designing a memory subsystem for maximum power and compute efficiency

The decode, or generation, phase of inference is fundamentally memory subsystem bound. It is not about peak bandwidth specs but rather how efficiently each kernel can utilize the entire memory subsystem. Modern reasoning and agentic workloads amplify this constraint by spending more end-to-end runtime in decode: long contexts, large KV caches, and interactive token generation make achieved memory bandwidth a critical performance lever.

Bar chart comparing Blackwell, Blackwell Ultra, and Rubin GPUs: Bandwidth rises from 8 TB/s to 22 TB/s on Rubin.
Figure 8. Memory bandwidth scaling from Blackwell to Rubin

Rubin addresses this with HBM4 combined with a highly efficient localized memory subsystem. HBM4 doubles interface width relative to HBM3e. Combined with a new memory controller, deep co-engineering with the memory ecosystem, and tighter compute-memory integration, this subsystem provides up to 22 TB/s of memory bandwidth: a 2.8x increase over Blackwell and Blackwell Ultra.

Rubin also provides up to 288 GB of HBM4 per GPU, increasing available on-package capacity relative to Blackwell. Capacity and bandwidth play different but complementary roles:

  • Capacity: Supports model residency, larger context windows, larger KV caches, and higher concurrency without unnecessary KV-cache offload.
  • Bandwidth: Supports the token-by-token generation phase, during which model weights and KV state must be moved rapidly enough to keep compute engines productive.
  • Memory subsystem: TMA and memory-locality strategies help software use the memory subsystem efficiently, supporting high achieved bandwidth for complex data layouts.

High-capacity, high-bandwidth HBM4 is critical for hosting multitrillion-parameter models, extending context length without KV cache offload, and supporting high-concurrency, long-horizon inference workloads.

How is NVIDIA designing for efficiency, scalability, and reliability?

Agentic AI infrastructure must optimize more than individual GPUs; it must make effective use of power, cooling, networking, and rack-level resources across the entire AI factory. NVIDIA Vera Rubin NVL72 extends the GPU architecture into an integrated, resilient rack-scale execution domain. This section examines how NVIDIA approaches power efficiency, operational scalability, and system-level reliability. 

Increasing the number of GPUs within the same power budget

Agentic AI makes energy efficiency an AI-factory problem, not simply a GPU power problem. At rack scale, Vera Rubin NVL72 integrates compute, networking, liquid cooling, power steering, and Intelligent Power Smoothing with energy storage into a single execution domain designed to maximize useful token output within a fixed power envelope.

GPU power chart shows capacitors covering spikes and filling idle-time valleys to maintain a steadier sustained AC power input.
Figure 9. GPU power plot showing impact of power smoothing techniques

Power demand can vary sharply during AI workloads, creating transient peaks that strand available capacity and reduce factory-level throughput. Vera Rubin power supplies use state-of-charge (SoC) Intelligent Power Smoothing to absorb these swings, reducing average power consumption by approximately 10% compared with previous-generation power-smoothing techniques and reducing 50 ms peak power by approximately 20%. By delivering a steadier power profile, the system can reduce sustained maximum power requirements, benefit supporting power infrastructure and the grid, and enable more compute within the same AI factory power budget. 

At the AI-factory level, NVIDIA DSX MaxLPS extends this approach across GPUs, racks, 45°C liquid cooling, and workloads, while DSX OS provides the operational layer for scheduling, lifecycle management, and health automation. Together, they are designed to recover stranded power and increase useful compute within a fixed megawatt envelope. DSX MaxLPS can enable operators to provision up to 40% more GPUs (Figure 10) within the same power budget at energy-efficient operating points, with minimal impact on workload performance.

Grid comparison shows unused AI factory capacity without DSX MaxLPS and up to 40% more GPU slots using DSX MaxLPS under the same power budget.
Figure 10. For the same power budget, DSX MaxLPS enables up to 40% more GPUs

Designing racks for resiliency and data center scalability

Vera Rubin NVL72 extends the Rubin GPU into a rack-scale execution domain for multitrillion-parameter and multi-rack agentic workloads. Its third-generation MGX rack architecture combines cable-free compute and switch trays, 45°C liquid cooling, dynamic rack-scale power steering, and Intelligent Power Smoothing to keep compute, networking, cooling, and power operating as one system. Hot-swappable NVLink switch trays and improved RAS capabilities further support resilient operation at scale.

This rack design is the foundation for pod-scale, multi-rack agentic systems, with open and flexible connectivity across NVLink and NVIDIA Spectrum-X Ethernet. 

Learn more

The NVIDIA Rubin GPU is designed for the execution patterns of agentic AI, where long-context reasoning, multistep generation, distributed MoE decode, and low-latency interaction must operate continuously at scale. Its compute, memory, networking, rack-scale, power, cooling, and software layers are carefully co-designed to keep more of the AI factory performing useful work rather than waiting on data, communication, or power constraints. The result is an architecture built to maximize agentic performance per watt—producing more useful tokens and completed AI work within a fixed power envelope. 

For a deeper look at the rack architecture and the broader NVIDIA Vera Rubin platform, see Inside the NVIDIA Vera Rubin Platform: Six New Chips, One AI Supercomputer. To learn more about NVIDIA Vera Rubin POD, see NVIDIA Vera Rubin POD: Seven Chips, Five Rack-Scale Systems, One AI Supercomputer.