NVIDIA Bolted a Diffusion Model Onto a Frozen LLM and Got 2.42x the Speed Without Retraining It

TL;DR

On July 1, 2026, NVIDIA published Nemotron-Labs-TwoTower, an open-weight diffusion language model that does something clever with an existing model instead of building a new one from scratch. It freezes a pretrained autoregressive model, the "context tower," and trains a second "denoiser tower" that generates whole blocks of tokens in parallel. At its default setting it hits 2.42x the wall-clock generation throughput of the autoregressive baseline while keeping 98.7% of its benchmark quality. The kicker: the denoiser was trained on about 2.1T tokens, roughly a twelfth of the 25T the backbone saw, so you can add fast parallel decoding to a checkpoint you already own without a full re-pretrain. The company whose entire business is selling you more compute just shipped a recipe for needing less of it, which tells you exactly how badly everyone wants generation to be faster.

What actually shipped

The model on Hugging Face is Nemotron-Labs-TwoTower-30B-A3B-Base-BF16. It is built on top of Nemotron-3-Nano-30B-A3B, an open-weight 30B hybrid Mamba-Transformer mixture-of-experts model. Because TwoTower keeps that backbone whole and stacks a second tower of equal size beside it, the package is about 60B total parameters (roughly 30B per tower), and each tower activates only about 3B parameters per token: 128 routable experts with 6 fired plus 2 shared. Each tower runs 52 layers, a mix of 23 Mamba-2, 6 self-attention, and 23 MoE blocks. It ships under NVIDIA's Nemotron Open Model License and is marked ready for commercial use.

The two-tower trick, and why it is not the usual diffusion LLM

Normal language models are autoregressive: they write one token, look at everything so far, then write the next, forever, strictly left to right. Diffusion models take the opposite bet. They start from a masked or noisy block and refine the whole thing at once over a fixed number of steps. If autoregressive decoding is typing a sentence one letter at a time, a diffusion block is a Polaroid: the entire picture develops together and sharpens with each pass. Parallel beats sequential when you can pull it off, which is the whole appeal.

The catch with pure diffusion LLMs has always been that they trail autoregressive models on quality and are expensive to train from zero. TwoTower's answer is to not throw the good model away. It splits the two jobs into two towers: a frozen autoregressive context tower that causally reads the clean prompt and every token committed so far, producing the KV cache and Mamba states, and a trainable denoiser tower that generates one block at a time via mask diffusion, using bidirectional attention inside the block and cross-attention back to the context tower. You keep the pretrained model's knowledge intact and bolt on a turbocharger that only has to learn the parallel-generation part.

The numbers hold up

Speed is easy; speed without wrecking quality is the hard part, and this is where TwoTower earns the headline. NVIDIA reports the diffusion model retains 98.7% of the autoregressive baseline's aggregate benchmark score. The per-task gaps are small and honest: on MMLU it scores 78.24 against the baseline's 78.56, on GSM8K 90.14 against 92.49, and it gives up the most on HumanEval code generation at 75.58 against 79.27. You are trading a few points of quality for well over double the tokens per second, and the trade is spelled out rather than hidden.

Why a builder should care

The interesting claim is not the specific model, it is the method. TwoTower shows that if you already have an autoregressive checkpoint, you can add diffusion-style parallel generation to it by training only a second denoiser network on a fraction of the original data budget, roughly 2.1T tokens against the backbone's 25T. That reframes fast decoding as a retrofit rather than a from-scratch research program. A lab, or a well-resourced team, that owns a good base model does not have to choose between "keep the quality" and "get diffusion speed." It can graft the second onto the first.

It also composes with the parts of the stack builders already care about. The context tower is frozen, so its KV cache and the pretrained weights are untouched; you are not risking a fine-tune eroding the base model's knowledge, because the base model literally does not move. The default operating point is a block of 16 tokens refined over 16 denoising steps, a knob you can turn to trade quality against speed for your own latency budget.

The honest caveats

This is a base model, not an instruction-tuned chat model, so it is a research artifact and a foundation to build on, not something to drop into a product this afternoon. The 2.42x is a wall-clock generation figure at a default configuration; real speedups depend on your batch sizes, hardware, and sequence lengths, and diffusion decoding has its own memory profile. The quality numbers, while close, are still a small step down from the autoregressive parent, and HumanEval shows code is where block diffusion currently strains most. And the benchmarks here are NVIDIA's own; independent reproduction on the released weights is the thing to watch next. Treat this as a strong, credible proof that the two-tower retrofit works, and wait for the community to stress it before betting a latency SLA on it.

Key Takeaways

• NVIDIA released Nemotron-Labs-TwoTower (July 1, 2026), an open-weight diffusion language model that freezes an autoregressive backbone and trains a separate denoiser tower to generate tokens in parallel blocks.
• It delivers 2.42x the generation throughput of the autoregressive baseline while retaining 98.7% of its aggregate benchmark quality.
• The denoiser was trained on about 2.1T tokens versus the backbone's 25T, so parallel decoding can be added to an existing checkpoint without a full re-pretrain.
• Built on Nemotron-3-Nano-30B-A3B: about 60B total params across two towers, roughly 3B active per token per tower, default 16-token blocks over 16 denoising steps.
• Quality gaps are small and disclosed (MMLU 78.24 vs 78.56, GSM8K 90.14 vs 92.49, HumanEval 75.58 vs 79.27); it is a base model with NVIDIA-reported benchmarks, so watch for independent reproduction.

Sources: Nemotron-Labs-TwoTower model card (Hugging Face), Nemotron-TwoTower paper (arXiv 2606.26493), MarkTechPost