Nemotron-Labs-TwoTower

Not to be confused with the two-tower retrieval model used in recommender systems.

Nemotron-Labs-TwoTower is an open-weight diffusion language model released by NVIDIA in mid-2026. It is the first public instantiation of TwoTower, a method described in the arXiv paper "Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context" (arXiv 2606.26493, first version June 25, 2026) by Fitsum Reda, John Kamalu, Roger Waleffe, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. ^[1] Instead of training a diffusion model from scratch or converting a single network to denoise, TwoTower takes two copies of a pretrained autoregressive model and gives them different jobs: a frozen AR context tower reads the prompt and every token already committed, exactly as the original model would, while a trainable diffusion denoiser tower fills in a block of masked tokens in parallel, cross-attending layer by layer to the context tower. ^[1][2] The released model is built on Nemotron 3 Nano 30B-A3B, a hybrid Mamba 2 and Transformer mixture-of-experts backbone with 30 billion total parameters, and the paper reports that after training the denoiser on approximately 2.1 trillion tokens it keeps 98.7 percent of the autoregressive baseline's aggregate benchmark quality while producing text with 2.42 times the wall-clock throughput at its default operating point. ^[1][2][3]

The weights are published on Hugging Face as nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16, a base (not instruction-tuned) checkpoint that ships both towers, roughly 60 billion parameters in total according to the model card, under what the card calls the NVIDIA Nemotron Open Model License. ^[3][4] The same checkpoint can also run as a plain autoregressive model using only the context tower, or in a "mock-AR" two-tower mode that predicts one token per step. ^[3] The name follows the "Nemotron-Labs" prefix NVIDIA has used since spring 2026 for research releases that sit beside the main Nemotron line, including Nemotron-Labs-Diffusion, Nemotron-Labs-Diffusion-Image, and Nemotron-Labs-3-Puzzle. ^[17][19][20]

Motivation: one network, two roles

Autoregressive (AR) language models decode one token per forward pass, which caps generation throughput when a single request is being served. Discrete diffusion language models, a line that runs from D3PM through MDLM to LLaDA and Block Diffusion, instead start from a fully masked block, predict every position in parallel, and refine the result over several steps. ^[2] The TwoTower authors argue that existing diffusion language models make one decoder do two different things at every denoising step: represent the clean tokens that are already known, and denoise the corrupted tokens that are being generated. In their words this "entanglement pulls the same set of weights in different directions, limiting their capacity to excel at either." ^[2]

The paper credits Arriola et al. (2025b), the Cornell group behind Block Diffusion (BD3-LM), with the observation that the two roles can be split across separate modules. Their encoder-decoder diffusion model, E2D2, uses an encoder to represent clean tokens and a lightweight decoder to iteratively denoise each block; the largest E2D2 experiment was at 1.7 billion parameters and trained both modules with tied weights. ^[2][11] TwoTower asks whether a full decoupling holds up at 30 billion parameters and on a modern hybrid architecture that mixes Mamba, attention, and MoE layers, and whether the context half can simply be the pretrained model, left untouched. ^[2]

Two further design decisions distinguish the approach from earlier adaptation work. First, the towers are connected at every layer: denoiser layer i cross-attends to the key-value cache of context-tower layer i, rather than receiving only the backbone's final hidden state. The authors argue that because both towers start from the same checkpoint, same-index layers "operate at comparable representation levels," which makes layer-aligned attention a natural pairing. ^[2] Second, the denoiser is conditioned on the diffusion timestep through adaptive layer normalization, a technique borrowed from image diffusion transformers (the paper cites DiT and PixArt-alpha) that the authors note is uncommon in masked diffusion language models. ^[2]

Architecture

TwoTower is described as a general recipe that can be applied to any pretrained AR language model. The released instantiation uses Nemotron-3-Nano-30B-A3B, a model NVIDIA pretrained from scratch on 25 trillion tokens, which has 52 layers: 23 Mamba-2 layers, 6 self-attention layers, and 23 mixture-of-experts layers. ^[2][3][9] Both towers are copies of this network. The context tower is left architecturally unchanged; the denoiser gets a small set of modifications, listed below. ^[2][3]

Scroll sideways for more →

Sources: paper Sections 2.1 and 2.3, model card. ^[2][3]

The bidirectional attention inside a block adds no parameters and, for standard dense attention kernels, does not change per-layer FLOPs; queries from the noisy block attend over a concatenated key-value sequence made of the context tower's layer-i cache for blocks 0 to b-1 and the denoiser's own layer-i keys and values for block b. ^[2] The adaLN modules are replicated on every tensor-parallel rank rather than sharded, which the authors say avoids extra communication for such small modules. ^[2]

The model card gives the parameter budget as roughly 60 billion in total (a 30B context tower plus a 30B denoiser tower) with about 3 billion active parameters per token per tower, and describes the MoE as "128 routable experts of which 6 are activated, with 2 shared experts." ^[3] The Hugging Face safetensors metadata for the repository counts 63,185,902,848 parameters, almost all in BF16, against 31,577,937,344 for the single-tower backbone repository. ^[4][7] Two details differ from NVIDIA's own Nemotron 3 Nano cards, which describe each MoE layer as "128 routed experts plus 1 shared expert, with 6 experts activated per token" and give 3.5 billion active parameters; the TwoTower card's "2 shared experts" and "about 3B" figures are reported here as written, and the discrepancy is not explained in either document. ^[3][8]

At inference time the design has a specific memory profile. The context tower's weights stay resident alongside the denoiser, so the fixed weight footprint roughly doubles relative to the backbone, but there is only one prefix cache: the context tower alone maintains KV and Mamba states across blocks, so the sequence-length-dependent memory scales like the AR baseline. ^[2] The model card says full two-tower diffusion inference places the towers on two GPUs with about 59 GB each for BF16 weights, and the bundled inference.py script states that the two-tower modes need "2 GPUs (118GB total)" while AR-only mode fits on one 80 GB GPU. ^[3][22]

Block-wise generation

Generation is block-wise autoregressive. Given a prompt, the context tower runs once to build per-layer KV pairs and final Mamba states. Each new block of S positions is initialized to [MASK] tokens and refined by the denoiser over up to T steps; once the block is clean it is committed, the context tower processes the committed tokens causally to update its caches, and the loop continues with the next block. ^[2] The paper's Algorithm 1 is this outer loop; Algorithm 2 is the single-block sampler used for the main results, which the authors call confidence unmasking. ^[2]

In confidence unmasking, every denoising step predicts all currently masked positions in parallel. Predictions whose confidence exceeds a threshold gamma are committed; the rest stay masked for a later step. A floor guarantees the block completes within T steps, so the number of tokens committed per step adapts to the model's confidence rather than following a fixed schedule. ^[2][3] The timestep fed to the denoiser at each step is the fraction of the block still masked. ^[2][3] The model card adds that predictions are constrained so the mask token itself is never emitted and already-decoded positions stay fixed. ^[3] The paper also mentions the standard predict-and-noise sampler from the block diffusion literature, which predicts a clean block and re-masks low-confidence positions according to the noise schedule, but does not use it for the headline results. ^[2]

The default operating point is gamma = 0.8 with S = 16 and up to 16 steps per block, in BF16 on two H100 GPUs. ^[2][3] Lowering gamma commits more tokens per step and raises throughput at some cost in quality; the paper says lower thresholds reach "throughput beyond 3x with larger quality loss," and that gamma = 0.8 was "the best balance we found." ^[2] Because the context tower keeps the backbone's autoregressive capability intact, the released checkpoint exposes three generation modes from one set of weights. ^[3]

Scroll sideways for more →

Sources: model card generation-modes table and Transformers examples. ^[3]

Training objective and recipe

Only the denoiser is trained. The context tower is run once per clean training sequence under its standard causal mask, with no gradient, to produce layer-aligned KV caches and the Mamba conv and SSM states after every clean block. The denoiser then takes the same sequence, applies masked-diffusion corruption, and processes all noisy blocks in a single forward pass: attention layers see past clean context plus their own noisy block, and Mamba layers fold the blocks into the batch dimension, with block 0 starting from a zero state and block b starting from the context tower's state after block b-1. The authors describe this batched formulation as exposing the denoiser to many block-level corruption patterns per optimizer step without enlarging the sequence batch. ^[2]

The objective is the masked diffusion framework of MDLM and related work (the paper cites Sahoo et al., Shi et al., and Ou et al.), applied per block. The sequence distribution is factorized block-autoregressively, each block conditional is parameterized by a masked diffusion process conditioned on the context caches, and the forward process replaces tokens with [MASK] under a linear schedule in which a clean token survives noise level t with probability 1 - t. The theoretical ELBO weight for that schedule is 1/t; the authors drop it "for stability" and optimize the plain mean negative log-likelihood over masked positions. ^[2] The training loop was implemented in Megatron-LM. ^[2][3]

Training follows the two-phase data curriculum of the Nemotron 3 Nano backbone: a broad-coverage phase-1 blend, then a phase-2 blend that upweights higher-quality and STEM-focused sources. Optimization uses BF16, AdamW, and a warmup-stable-decay learning-rate schedule with peak learning rate 1e-4 and final learning rate 1e-6, reset at phase boundaries. In the current implementation the Mamba chunk size is matched to the block size so that the existing Mamba kernel exposes states at block boundaries. ^[2][3]

Scroll sideways for more →

Source: paper Section 3.5 and the model card's training methodology section. ^[2][3]

The total token count is stated inconsistently across NVIDIA's own documents. The abstract, the results section, and the model card all give approximately 2.1 trillion tokens for the released checkpoint, a figure the paper contrasts with the 25 trillion tokens used to pretrain the backbone. ^[1][2][3] The training section of the paper, describing the two-phase recipe, says "across both phases, we train on roughly 1.4T tokens." ^[2] The paper does not say how the two figures relate; the released checkpoint adds a third training stage, which may account for the difference, but that is an inference rather than something the authors state. The model card's "Model Dates" field gives September 2025 to April 2026 for the work, and the pre-training data cutoff inherited from the backbone is June 25, 2025. ^[3]

The denoiser's data comes from the same sources as the backbone. The card lists nine public Nemotron datasets (Nemotron-Pretraining-Code-v1 and v2, Nemotron-CC v2 and v2.1, Nemotron-Pretraining-SFT-v1, Nemotron-CC-Math-v1, Nemotron-Pretraining-Specialized-v1, Nemotron-CC-Code-v1, and the Nemotron-Pretraining-Dataset-sample) and describes the training data as a roughly 2.1T-token subset of the backbone's two-phase blend, collected through a mix of automated, human, and synthetic methods. ^[3]

Results for the released checkpoint

All evaluations in the paper compare against the autoregressive Nemotron-3-Nano-30B-A3B checkpoint that initialized both towers, using base-model checkpoints taken after long-context pretraining and before instruction tuning, reinforcement learning, or alignment. Everything runs in BF16 on two H100 GPUs, and throughput is measured as wall-clock time to produce the final answer under the same inference setup, reported as a speedup over the AR baseline. ^[2] The headline numbers, 98.7 percent of aggregate quality retained and 2.42x throughput, are NVIDIA's own measurements at gamma = 0.8 and S = 16. ^[1][2][3]

The model card gives per-benchmark results for the AR baseline and the diffusion mode. The card's source Markdown still carries an editorial comment asking to "fill in final released-checkpoint numbers" and to confirm the AR column, so the table should be read as NVIDIA's published card values rather than an independently reproduced result. ^[3]

Scroll sideways for more →

Source: model card benchmark table; NVIDIA-reported. ^[3]

The paper's Figure 2, which the card reproduces as table_comparison.png, aggregates the same runs by category. The figure labels the two systems "Nemotron 3 Nano Base" and "Nemotron 3 Nano Diffusion" and reports throughput for the generative tasks excluding MMLU. ^[2][3]

Scroll sideways for more →

Source: values read from the published Figure 2 chart; NVIDIA-reported. ^[2][3]

The authors summarize the pattern as general knowledge staying within about one point of the baseline, code and math showing "modest degradation," and commonsense and multilingual results being "recovered or improved." They stress that the recovery comes from adaptation rather than re-pretraining: the model starts from an off-the-shelf AR checkpoint and trains only the denoiser. ^[2]

Ablations

The paper's ablations were run on an internal Nemotron-3-Nano-30B-A3B base checkpoint rather than the final released one, so their absolute numbers are not directly comparable with the tables above. Accuracy is reported for three groups: generation, code, and math. ^[2]

Design choices (Table 1, block size 32)

Scroll sideways for more →

Source: paper Table 1. ^[2]

The bidirectional Mamba variant, which runs the pretrained Mamba-2 weights left-to-right and right-to-left from zero states and averages the outputs, left generation essentially unchanged while lowering code and math, and roughly doubles per-layer SSM compute in the denoiser, so the final design keeps Mamba causal and relies on bidirectional attention for in-block context. AdaLN time conditioning lifted all three groups for 1.5 million added parameters, which the authors take as evidence that the denoiser benefits from knowing the current noise level. Continuing on the phase-2 blend added further gains, mirroring the backbone's own curriculum. ^[2]

Tower decoupling (Table 2, after about 167B phase-1 tokens)

Scroll sideways for more →

Values are relative accuracy change in percentage points from the reference. Source: paper Table 2. ^[2]

This table is the paper's evidence for the core design decision. Continuing to train the backbone itself on the same budget hurt AR accuracy noticeably, tying the two towers under a joint objective was "substantially worse under both AR and diffusion decoding modes," and the decoupled design with a frozen context tower gave the smallest drop. The authors conclude that the context tower should stay frozen and a separate denoiser should be adapted. ^[2]

Training block size (Table 3)

Scroll sideways for more →

Training and inference use the same block size in each row; throughput is the wall-clock speedup over the AR baseline for that run. Source: paper Table 3. ^[2]

Smaller training blocks improve quality: within phase 1, going from S = 128 to S = 32 raised the three groups from 69.59/63.56/75.73 to 72.94/68.64/80.57, and after phase-2 continuation S = 16 reached 77.10/74.56/85.45. S = 8 added only marginal generation and math quality while dropping throughput to 1.71x, so S = 16 was chosen as the default because it "captures most of the quality gain from smaller blocks while retaining a clear speed advantage." ^[2]

Sampling block size (Table 4, checkpoint trained at S = 16)

Scroll sideways for more →

Source: paper Table 4. ^[2]

The asymmetry is the notable result. Sampling with blocks larger than the training block size collapses generation-heavy tasks: HumanEval falls from 76.40 at S = 16 to 19.85 at S = 64, and GSM8K and MATH-500 fall to 2.20. Multiple-choice MMLU, which needs almost no generation, barely moves. Sampling with smaller blocks holds up and even slightly improves MMLU, GSM8K, MATH-500, and the multilingual average, but smaller blocks require more frequent context-tower updates and therefore lower throughput, which is why S = 16 remains the default. ^[2]

Sampling dynamics

Section 3.6 of the paper studies how the released checkpoint actually spends its denoising steps at gamma = 0.8 and S = 16, using 100 problems per multi-token generative benchmark and recording the trace for every answer-producing block. Three regularities are reported. Block completion is adaptive: on GSM8K and MMLU-Pro many blocks finish in the first few steps, with MMLU-Pro showing a longer tail, and the appendix shows MGSM and MBPP-Sanitized often finishing blocks immediately while code and reasoning tasks sustain longer tails. Commitments are front-loaded in time: the first step commits the most tokens on average and the count drops quickly, which the authors offer as the explanation for how a multi-step sampler can still beat one-token-per-step decoding on wall-clock time. And commitments follow a left-to-right order within a block: earlier positions are usually committed first, producing an "upper-left triangular pattern" in the position-by-step histogram. ^[2]

The authors' explanation for the last effect is architectural. The context tower is fully causal, and the denoiser keeps the same unidirectional Mamba structure while adding bidirectional attention only inside the block; because the backbone has 23 Mamba layers and just 6 attention layers, they suggest that a strong left-to-right inductive bias inherited from pretraining dominates the sampling behavior. ^[2]

Release history and artifacts

The Hugging Face repository was created on April 11, 2026, and its public commit log shows the model evolving from a two-tower autoregressive checkpoint into the mask-diffusion release over the following weeks. ^[6] The card lists two versions: v1.0, a two-tower AR (mock-AR) checkpoint, and v1.1, which enabled block-wise mask-diffusion generation with the time-conditioned denoiser, bidirectional in-block attention, and context-seeded chunk-scan Mamba-2. ^[3]

Scroll sideways for more →

The arXiv listing's comments field says "Code and model weights available at" the Hugging Face collection, and the paper's conclusion says NVIDIA releases "weights, training code, and recipe in the Nemotron-Labs-TwoTower collection." ^[1][2] As of September 6, 2026, that collection contains a single item, the model repository. ^[5] The repository holds the weights in 24 safetensors shards, the modeling code (modeling_nemotron_twotower.py, about 45 KB, plus the NemotronH backbone files), an inference.py example, the paper PDF, tokenizer files, and Model Card++ subcards for bias, explainability, privacy, and safety. ^[4][22] The modeling file's header describes the checkpoint layout (separate context_tower.* and denoiser_tower.* weights, a timestep embedder, a timestep MLP, and per-layer modulation tables) and the three generation entry points. ^[22] This wiki did not locate a separate training-code repository on GitHub: the Megatron-LM main tree contains no TwoTower paths, and several plausible NVIDIA repository names return not-found responses, so "training code" in the release statements appears to refer to the recipe described in the paper and the Megatron-LM implementation the paper cites rather than to a standalone published training repository. Press reports that NVIDIA "published open weights and training code" should be read with that in mind. ^[13]

The model card requires trust_remote_code=True and provides a place_towers_on_devices("cuda:0", "cuda:1") helper to split the towers across two GPUs, with generate_mask_diffusion(), generate_mock_ar(), and generate_ar() as the three entry points; the diffusion example uses block_size=16, steps_per_block=16, mask_token_id=3, temperature=0.1, and confidence_threshold=0.8. ^[3] Supported hardware per the card is the NVIDIA H100 80 GB and A100, on Linux, through Hugging Face Transformers. Maximum input and output sizes are listed as 128K tokens each. ^[3]

License

The license is described slightly differently in two places on the same card. The card's structured metadata (license: other, license_name: nvidia-open-model-license) links to the NVIDIA Open Model License page, while the card's badge and its "License/Terms of Use" section say use "is governed by the NVIDIA Nemotron Open Model License Agreement" and link to that agreement instead. ^[3][4] The card states that "this model is ready for commercial use." ^[3] The backbone card for NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 uses the Nemotron Open Model License in both its metadata and its prose. ^[7]

Usage signals

As of September 6, 2026, the Hugging Face API reported 817 downloads (a rolling 30-day count) and 142 likes for the repository. ^[4] The repository's community tab held four open threads: a question about the architecture and checkpoint size (July 3), a bug report that the CPU forward path crashes on cache handling (July 5), a request for a DOI (July 8), and a pull request titled "Fix _mdlm_forward: unmasked positions get all-zero probabilities" (July 10). ^[21] Independent ports appeared within days of the July 1 rename: a repository for running the model on Apple Silicon with MLX, created July 2; an AGPL-licensed project for running it on CPUs (x86_64, arm64, and ppc64le) without an NVIDIA GPU, created July 5; and a project splitting the two towers across two DGX Spark systems with NVFP4-quantized experts, whose own description claims 38.9 tokens per second, a figure this wiki has not verified. ^[23][24][25]

Relationship to other diffusion language models

The paper positions TwoTower inside the masked discrete diffusion lineage and cites D3PM (Austin et al., 2021), MDLM (Sahoo et al., 2024), the simplified masked diffusion of Shi et al. (2024), the "absorbing discrete diffusion" analysis of Ou et al. (2024), LLaDA (Nie et al., 2025), Block Diffusion (Arriola et al., 2025a), and E2D2 (Arriola et al., 2025b) as its nearest neighbors. ^[2] Block Diffusion introduced the block-autoregressive factorization that TwoTower adopts, interpolating between fully autoregressive and fully parallel generation while keeping a KV cache across blocks. ^[10] E2D2 is the closest architectural relative: it too separates clean-token representation from denoising, but with a lightweight decoder and tied weights at up to 1.7B parameters, whereas TwoTower uses two full 30B towers, freezes one of them, and connects them at every layer. ^[2][11] For the broader field, including Mercury from Inception Labs, Gemini Diffusion, Dream 7B, and the LLaDA 2.0 family, see the diffusion language models survey and discrete diffusion.

TwoTower's specific claim to novelty, as the authors frame it, is threefold: it is the first demonstration they know of that a fully decoupled context-and-denoiser design holds at 30 billion parameters on a hybrid Mamba-attention-MoE backbone; the frozen context tower means the pretrained model's AR ability is preserved rather than traded away, so one checkpoint serves both decoding regimes; and layer-aligned cross-attention gives the denoiser multi-scale access to the backbone rather than a single summary vector. ^[2] The paper also observes that keeping the context tower's LM head makes it usable as a verifier for speculative decoding, likelihood evaluation, or AR scoring, though it reports no speculative-decoding experiments. ^[2]

Other Nemotron-Labs releases

The "Nemotron-Labs" prefix appeared on several NVIDIA research releases within weeks of TwoTower, from different author groups. Nemotron-Labs-Diffusion (arXiv 2607.05722, July 7, 2026; authors led by Yonggan Fu with Pavlo Molchanov as last author) is a separate tri-mode language model family at 3B, 8B, and 14B parameters, trained with a joint AR-diffusion objective so that one set of weights can decode autoregressively, by diffusion, or in a self-speculation mode in which diffusion drafts and AR verifies; its Hugging Face collection lists base, instruct, and vision-language variants. ^[17][18] It should not be confused with TwoTower, which keeps two separate towers and does not train the AR half. Nemotron-Labs-Diffusion-Image (arXiv 2606.29814, June 29, 2026) applies masked discrete diffusion to high-resolution image synthesis, and Nemotron-Labs-3-Puzzle-75B-A9B (arXiv 2607.04371, July 5, 2026) compresses hybrid MoE models. ^[19][20] NVIDIA's Hugging Face account also carries Nemotron-Labs-Audex and Nemotron-Labs-Elastic collections. ^[5] The TwoTower repository itself was renamed to carry the prefix on July 1, six days after the paper first appeared under the name "Nemotron-TwoTower." ^[1][6]

Reception

Coverage concentrated in the first week of July 2026. MarkTechPost's Asif Razzaq (July 1) summarized the paper and card, reproducing the per-benchmark table and the "one checkpoint runs diffusion, mock-AR, and AR decoding modes" framing. ^[12] TechTimes' Jerry Owens (July 2) emphasized the training economics, writing that the method "shifts the economics of the diffusion LM transition from a cliff to a ramp" because a lab with an existing AR checkpoint can add parallel generation by training only a second network, and noted the 3.7-point HumanEval drop as the clearest cost; the piece also stated that the released checkpoint is a base model that "has not been instruction-tuned." ^[13] Japan's Zaikei Shimbun (July 3) and China's IT Home, syndicated by Sina Technology (July 3), carried the same 2.42x and 98.7 percent figures; IT Home described the release as a 60B-parameter dual-tower model with 128 routable experts. ^[14][15] TechTimes returned to the topic on July 10 in a piece by Mireya Ramsey about the separate Nemotron-Labs-Diffusion release, which it framed as showing that "the AR vs. diffusion dichotomy was a training artifact." ^[18]

The paper circulated again on social media in September 2026. On September 5, Avi Chawla posted an explainer to X that opened "NVIDIA might just have solved the biggest tradeoff in LLMs," described the frozen context tower as one where "the model's intelligence stays perfectly intact," and closed with "Weights and code are all open-source." ^[16] The "solved the biggest tradeoff" framing is Chawla's, not NVIDIA's; the paper's own language is that the model "preserves most of the autoregressive baseline's quality" and that masked diffusion "can serve as a practical decoding adaptation for large pretrained AR models." ^[2][16] The post's summary bullets (2.42x throughput, 98.7 percent quality, a 30B hybrid Mamba-Transformer MoE backbone, about 2.1T training tokens against 25T for pretraining) match the paper. ^[1][16]

Limitations and open questions

Several caveats follow directly from the sources. The released model is a base checkpoint with no instruction tuning or alignment; the paper says NVIDIA plans to add post-trained TwoTower models to the same collection "in future updates," which is stated as a plan and had not happened as of September 6, 2026, when the collection still held one repository. ^[2][5] Full diffusion decoding needs two 80 GB GPUs because both towers must be resident, which the authors acknowledge increases the fixed weight footprint even though cache memory scales like the baseline. ^[2][3] Code and math tasks lose a few points at the default operating point (HumanEval 79.27 to 75.58, MATH-500 84.40 to 80.60), and the block-size study shows that sampling with blocks larger than the training block size is catastrophic for generative tasks, so the S = 16 checkpoint should not be run at S = 32 or 64. ^[2][3] The 2.42x throughput figure is NVIDIA's measurement on two H100s in BF16 with its own inference setup; no independent replication had been published as of this writing, and the card's own Markdown still contains reviewer notes asking to confirm the AR baseline column and the evaluation harness. ^[2][3] Finally, the training-token budget is reported as roughly 1.4T in the paper's method section and approximately 2.1T everywhere else, a discrepancy the documents do not reconcile. ^[2][3]

References

1. ^Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context (abstract page) - arXiv (Fitsum Reda, John Kamalu, Roger Waleffe, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro), v1 June 25, 2026, v2 June 29, 2026.
2. ^Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context (full text, v2) - arXiv HTML (Reda et al.), June 29, 2026.
3. ^nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16 model card - Hugging Face (NVIDIA), last modified July 1, 2026.
4. ^Hugging Face API record for nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16 - Hugging Face, fetched September 6, 2026.
5. ^Nemotron-Labs-TwoTower collection - Hugging Face (NVIDIA), last updated August 11, 2026; fetched September 6, 2026.
6. ^Commit history of nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16 - Hugging Face, April 11 to July 1, 2026; fetched September 6, 2026.
7. ^nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 model card - Hugging Face (NVIDIA), released December 15, 2025; fetched September 6, 2026.
8. ^nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 model card - Hugging Face (NVIDIA), fetched September 6, 2026.
9. ^Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoning - arXiv (NVIDIA), December 2025.
10. ^Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models - arXiv (Marianne Arriola, Aaron Gokaslan, Justin T. Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov), March 12, 2025.
11. ^Encoder-Decoder Diffusion Language Models for Efficient Training and Inference - arXiv (Marianne Arriola, Yair Schiff, Hao Phung, Aaron Gokaslan, Volodymyr Kuleshov), October 26, 2025.
12. ^NVIDIA Releases Nemotron-Labs-TwoTower: an Open-Weight Diffusion Language Model Built on a Frozen Autoregressive Nemotron-3-Nano-30B-A3B Backbone - MarkTechPost (Asif Razzaq), July 1, 2026.
13. ^NVIDIA Diffusion LLM Hits 2.42x Throughput Without Retraining: Nemotron TwoTower Released - TechTimes (Jerry Owens), July 2, 2026.
14. ^NVIDIA、再学習不要でスループットを2.42倍にする拡散LLM「Nemotron TwoTower」を公開 - Zaikei Shimbun, July 3, 2026.
15. ^英伟达开源 TwoTower AI 模型：保留 98.7% 质量，Token 生成提速 2.42 倍 - Sina Technology (IT Home), July 3, 2026.
16. ^Post by Avi Chawla (@_avichawla) on X - X (Avi Chawla), September 5, 2026.
17. ^Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding - arXiv (Yonggan Fu et al.), July 7, 2026.
18. ^Diffusion LLM Learns to Be Its Own Draft Model: NVIDIA Releases Tri-Mode Open Weights - TechTimes (Mireya Ramsey), July 10, 2026.
19. ^Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis - arXiv (Shufan Li, Greg Heinrich, Hanrong Ye, Yonggan Fu, Aditya Grover, Jan Kautz, Pavlo Molchanov), v1 June 29, 2026, v2 July 15, 2026.
20. ^Nemotron-Labs-3-Puzzle-75B-A9B: Compressing Hybrid MoE LLMs - arXiv (Akhiad Bercovich et al.), v1 July 5, 2026, v2 July 7, 2026.
21. ^Community discussions for nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16 - Hugging Face, July 3 to July 10, 2026; fetched September 6, 2026.
22. ^modeling_nemotron_twotower.py and inference.py in the model repository - Hugging Face (NVIDIA), fetched September 6, 2026.
23. ^PipeNetwork/nemotron-twotower-mlx - GitHub, created July 2, 2026.
24. ^Scottcjn/nemotron-twotower-cpu - GitHub, created July 5, 2026.
25. ^drowzeys/Keys-NVIDIA-Two-Tower-Diffusion--dual-dgx-spark - GitHub, created July 5, 2026.