NVIDIA Dynamo Accelerates Multimodal AI Inference

NVIDIA has introduced encode-prefill-decode (EPD) disaggregation to its open-source Dynamo inference framework. This technique separates vision encoding from LLM prefill and decode stages, delivering up to a 5x faster time to first token (TTFT) and a 7x faster end-to-end response time for image-heavy prompts. The optimization is highly effective for short-to-medium outputs and quantized mixture-of-experts models where vision processing bottlenecks the pipeline.

Practitioners can deploy EPD using three topologies: aggregated serving, colocated encoder workers, or disaggregated encoder workers. In heterogeneous setups, developers can run encoder workers on a lower-cost GPU tier, like two NVIDIA RTX 6000D GPUs, while reserving four NVIDIA GB200 GPUs for prefill and decode stages, connecting them via the NVIDIA Inference Transfer Library (NIXL). Benchmarks using the Qwen3.5 122B A10B NVFP4 model with ten images per request and an output sequence length (OSL) of 1024 showed a 58% TTFT drop for colocated setups and a 50% drop for heterogeneous setups, while serving 70% more traffic under an inter-token latency service level objective under 100 milliseconds.

The benefits scale with model size and precision. In tests using Qwen3.5 4B, 9B, and 27B, colocated EPD delivered 2.62x, 1.50x, and 0.65x the goodput of aggregated serving, respectively. Quantizing active weights to NVFP4 while keeping the vision encoder in BF16 increased colocated goodput from 1.78x to 2.64x. Under a 50:50 mix of text and image traffic at OSL 128, EPD reduced mean TTFT for text requests by 42.2% (from 92.3 to 53.3 milliseconds) and for image requests by 30.8% (from 289.9 to 200.6 milliseconds) by eliminating head-of-line blocking.

However, EPD is not always ideal. When OSL increased from 128 to 2,048 with five images, the heterogeneous end-to-end latency gain narrowed from 20.3% to 5.2%, and colocated performance regressed by 2.5% due to GPU contention. To further optimize, Dynamo supports parallel media decoding, which cut mean encoder latency by 26% (from 281.3 to 207.0 milliseconds) in a 30x256 image benchmark on a single GB200. Other features include an embedding cache and multimodal KV routing.