NVIDIA Introduces a 4-Bit Pretraining Methodology Using NVFP4, Validated on a 12B Hybrid Mamba-Transformer at 10T Token Horizon

Pretraining frontier-scale LLMs in FP8 is now standard practice, but moving to 4-bit floating point has remained an open research problem because narrower formats compress dynamic range and amplify quantization error at long token horizons. A new research from NVIDIA describes a pretraining methodology built around NVFP4, a 4-bit microscaling format supported natively by Blackwell Tensor Cores, and validates it by pretraining a 12-billion-parameter hybrid Mamba-Transformer on 10 trillion tokens. The research team state this is the longest publicly documented training run in 4-bit precision to date. The resulting model attains 62.58% on MMLU-Pro 5-shot versus 62.62% for the FP8 baseline, and is supported in NVIDIA’s Transformer Engine.

What NVFP4 Actually is

To understand why NVFP4 is important, it helps to revisit how microscaling formats work. In a microscaling (MX) format, a contiguous block of low-precision elements shares a single scale factor, which is used to map the block back into a wider numerical range during the matrix multiply. MXFP4 uses 32-element blocks where each element is stored as E2M1 — 1 sign bit, 2 exponent bits, 1 mantissa bit — encoding only the values ±0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, and ±6. Block scale factors are stored in UE8M0, which restricts them to powers of two.

NVFP4 changes three things. First, the block size drops from 32 to 16 elements, narrowing the dynamic range each scale has to cover. Second, block scale factors are stored in E4M3 rather than UE8M0, trading exponent range for mantissa precision so the per-block amax (absolute maximum) can be mapped much closer to the FP4 maximum representable. Third, NVFP4 adds a second scaling level: an FP32 per-tensor scale that remaps values so the E4M3 block scales themselves stay in range. The result is that at least 6.25% of values in each block — the per-block amax — are represented at near-FP8 precision, while the remainder sit in FP4.

On NVIDIA Blackwell, FP4 GEMMs run at 4× BF16 throughput on GB200 and 6× on GB300, which translates to roughly 2× and 3× speedups over FP8. Operand memory footprint is approximately halved compared to FP8.

What’s Quantized — and What Isn’t

Only the GEMMs inside linear (fully-connected) layers Fprop, Dgrad, and Wgrad actually run in NVFP4. Embeddings, the output projection head, normalization layers, non-linearities, and all attention components (softmax and the query-key and attention score-value batched GEMMs) stay in BF16 or FP32. Model weights, weight gradients used for accumulation across microbatches and data-parallel replicas, and optimizer states are kept in FP32. Tensor parallel reductions run in BF16.

The Four-Part Training Methodology

Quantizing every linear-layer GEMM to NVFP4 with default settings (1×16 block scaling everywhere, round-to-nearest-even on every tensor, no transforms) diverges early in training. NVIDIA’s approach stabilizes it with four components, and ablation studies on the 12B model show each is necessary.

Selective high precision: Linear layers in the first two and the final eight of the 62 blocks (about 16% of all linear layers) are kept in BF16. Ablations indicated that the final blocks are the sensitive ones because they require more dynamic range than FP4 provides; keeping only the final four blocks in BF16 was also enough for stable convergence.

Random Hadamard Transforms (RHT): Outliers in weight gradients are spread into an approximately Gaussian distribution by multiplying the input tiles with a 16×16 Hadamard matrix combined with a random ±1 sign vector. Because the orthogonal transforms cancel inside the dot-product, no math correction is needed in the GEMM. The d=16 size was chosen empirically: d=4 hurt convergence, d=128 gave similar results. RHT is applied only to the inputs of the weight-gradient (Wgrad) GEMM, and a single random sign vector is shared across all linear layers. Randomization itself was a no-op at the 1.2B scale but measurably improved the 12B run.

Two-dimensional (2D) block scaling for weights: Standard NVFP4 scales 1×16 blocks along the dot-product dimension. Because the backward pass transposes the weight tensor, the forward and backward passes end up with different quantized weights, breaking the chain rule. NVIDIA’s fix is to scale weights in 16×16 blocks so the same quantized representation is used in both passes. Activations and gradients keep 1×16 scaling, since they are less sensitive to this inconsistency.

Stochastic rounding on gradients: Round-to-nearest-even introduces systematic bias when applied to gradient tensors. Stochastic rounding rounds probabilistically based on distance to the two nearest representable values, removing that bias. The research team explicitly notes in research paper that stochastic rounding is detrimental when applied to forward-pass tensors, so it is restricted to gradients.

Results on the 12B Hybrid Mamba-Transformer

The 12B model uses the Nemotron-Nano-12B-v2-Base architecture — 62 blocks (6 Self-Attention, 28 FFN, 28 Mamba-2), hidden dimension 5120, FFN dimension 20480 — trained with a Warmup-Stable-Decay schedule (constant LR through 80% of training, decay over the final 20%), batch size 736, sequence length 8192. The FP8 reference baseline follows the DeepSeek-V3 methodology: E4M3 elements, 128×128 weight blocks, 1×128 activation and gradient blocks, with the first block and last two blocks kept in BF16.

NVFP4 validation loss stays within 1% of the FP8 baseline during the stable phase and widens to slightly above 1.5% during decay. Downstream accuracy is comparable across most benchmarks: MMLU 76.57% vs 77.36%, GSM8K CoT 92.27% vs 89.08%, MATH 81.48% vs 83.32%, AGIEval English CoT 70.31% vs 67.01%. Coding shows the largest gap — HumanEval+ 57.43% vs 59.93%, MBPP+ 55.91% vs 59.11% — which the research team attributes partly to noisy final-checkpoint evaluation. The research team also documents a precision-switching technique: transitioning the forward pass from NVFP4 to BF16 starting at 8.2T tokens (about 18% of the schedule) reduced relative loss error from 1.5% to 0.5%.

NVFP4 vs MXFP4

On a separate 8B hybrid Mamba-Transformer trained on 1T tokens, NVFP4 reached a relative loss error of about 1.5% versus BF16, while MXFP4 stayed near 2.5%. To close the gap, MXFP4 required 1.36T tokens to match the NVFP4 1T-token loss — a 36% token overhead. The research team attributes the difference to NVFP4’s smaller block size and E4M3 scales, which preserve more of the FP4 dynamic range than MXFP4’s power-of-two UE8M0 scales (which can waste up to one binade and the ±4, ±6 samples in the worst case).

Marktechpost’s Visual Explainer

Key Takeaways

NVIDIA’s research team pretrained a 12B hybrid Mamba-Transformer on 10T tokens in NVFP4 — the longest publicly documented 4-bit training run — matching FP8 on MMLU-Pro at 62.58% vs 62.62%.

NVFP4 uses 16-element blocks with E4M3 scales plus an FP32 per-tensor scale, preserving the ±4 and ±6 samples that MXFP4’s 32-element UE8M0 design can lose to power-of-two rounding.

Four techniques are required for convergence — none are optional: ~16% of linear layers in BF16, 16×16 Random Hadamard Transforms on Wgrad inputs, 2D 16×16 weight scaling, and stochastic rounding on gradients only.

Only linear-layer GEMMs run in NVFP4 — attention, embeddings, normalization, non-linearities, master weights, gradients, and optimizer states all stay in BF16 or FP32.

On an 8B model, MXFP4 needed 1.36T tokens (36% more) to match NVFP4’s loss at 1T tokens, while FP4 GEMMs deliver 2× FP8 throughput on GB200 and 3× on GB300.

Check out the Paper here. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us