NVIDIA NIM Speeds Up Nemotron 3 Ultra by 2.5x

NVIDIA has detailed a series of full-stack optimizations in its NIM 2.0.12 serving stack that significantly increase user capacity for its Nemotron 3 Ultra model. Running on a four-GPU Blackwell B200 system, the optimized microservice achieved a system throughput of 1,997 tokens per second, compared to just 718 tokens per second on a baseline stack without these optimizations. This represents a 2.5x performance increase while maintaining a target of 50 tokens per second per user.

The benchmark evaluated an agentic workload characterized by a 64K context window, 400 output tokens, 76 percent key-value cache reuse, and a 20-millisecond time-to-first-token latency. To achieve these gains, NVIDIA combined several engineering techniques. These include autotuned mixture-of-experts and Mamba kernels, tensor parallelism across the four B200 GPUs, prefix caching, partial-prefix matching, and multi-token prediction speculative decoding.

For practitioners, these optimizations address the high computational costs of deploying large language models like the 550-billion-parameter Nemotron 3 Ultra, specifically the nvidia/nemotron-3-ultra-550b-a55b profile. Instead of manually configuring complex runtime settings, developers can deploy a pre-validated container through NVIDIA AI Enterprise. They can also use the NVIDIA AIPerf tool to replay their own traffic and find the optimal Pareto point that satisfies their latency service-level objectives.

The optimized microservice is available to download via the NVIDIA GPU Cloud registry. To deploy the specific configuration tested for agentic workloads on the four-GPU B200 setup, developers can select the throughput-optimized profile named vllm-nvidia-b200-nvfp4-tp4-pp1-throughput-90.0 and enable speculative decoding.