flashattentionfp4-quantizationblackwelldirect-pgpu-kernelsllm-inferencehardware-acceleration

Hardware-Aware FP4 FlashAttention-4: Direct-P Guide

October 9, 2026

Hardware-Aware FP4 FlashAttention-4 Architecture: Direct-P Log-Score-to-E2M1 Mapping on NVIDIA Blackwell Tensor Cores

Modern deep learning accelerators have reached an inflection point where floating-point numerical precision and kernel scheduling must be co-designed together. In cutting-edge generative video pipelines, reasoning-dense foundation models, and high-concurrency web data extraction platforms powered by AI Fetchly, self-attention accounts for a massive fraction of overall GPU runtime.

With the arrival of NVIDIA’s Blackwell architecture (such as the B200 and GB200 systems), hardware compute capabilities have leaped forward: Blackwell’s native 4-bit floating-point (FP4) Tensor Cores offer double the raw theoretical throughput of 8-bit floating-point (FP8) operations and four times that of 16-bit brain floating-point (BF16). However, while General Matrix Multiply (GEMM) layers—such as Feed-Forward Networks (FFNs) and linear projections—natively embrace 4-bit quantization, applying FP4 to FlashAttention has historically remained an elusive challenge.

The barrier lies in the mathematical asymmetry of self-attention. While Query-Key ($Q K^\top$) and Probability-Value ($P V$) steps are standard matrix multiplications, the intermediate Softmax operator ($P = \text{softmax}(S)$) requires evaluating non-linear exponentials across rows. In conventional kernel implementations, evaluating high-precision floating-point exponentials and dynamically quantizing the resulting probabilities to 4-bit E2M1 format stalls the hardware pipeline, completely erasing the $2\times$ compute advantage of FP4.

To conquer this bottleneck, Graphcore Research introduced Hardware-Aware FP4 FlashAttention-4: Direct-P (arXiv:2609.04105). Direct-P redesigns the entire probability path inside the tiled attention loop: replacing intermediate floating exponentials with a direct log-score-to-E2M1 map, deploying a hybrid combination of NVFP4 for $QK$ and MXFP4 for $PV$, and accumulating denominators directly from the represented 4-bit probabilities. The result is a breakthrough kernel achieving up to 2.13x higher forward throughput than BF16 FlashAttention on Blackwell B200 GPUs.

In this comprehensive technical guide, we break down the hardware realities of Blackwell FP4 Tensor Cores, examine why naive quantization fails in FlashAttention, dissect the three core pillars of Direct-P, construct a functional PyTorch simulation, and evaluate empirical speedups.

The Blackwell FP4 Attention Bottleneck: Why Naive Approaches Fail

To understand why previous attempts at 4-bit attention struggled, we must examine the physical pipeline of tiled FlashAttention and the numerical anatomy of Blackwell’s FP4 data formats.

The Tiled FlashAttention Pipeline

In standard FlashAttention, input matrices $Q$, $K$, and $V$ are partitioned into blocks that fit within high-speed GPU on-chip Shared Memory (SRAM). For a given tile of queries $Q_i \in \mathbb{R}^{B_r \times d}$ and keys $K_j \in \mathbb{R}^{B_c \times d}$, the kernel executes two consecutive GEMM operations interleaved with Softmax:

GEMM 1 (Score Computation): $S_{ij} = \frac{Q_i K_j^\top}{\sqrt{d}}$
Softmax Rescaling: $P_{ij} = \exp(S_{ij} - m_i)$, where $m_i = \max(S_{ij})$ tracks the running row maximum.
GEMM 2 (Context Aggregation): $O_i = P_{ij} V_j$

On NVIDIA Hopper (H100) GPUs, FlashAttention-3 achieved extreme speedups by overlapping GEMM 1 and GEMM 2 using Asynchronous Warp Specialization in FP16 or FP8. However, on Blackwell GPUs, accelerating this loop with 4-bit precision exposes two crippling hardware constraints:

1. The Critical Path Stalling of Intermediate Exponentials

In a naive FP4 attention port, after GEMM 1 computes raw scores $S_{ij}$, the Special Function Units (SFU) must calculate transcendental exponentials $\exp(S_{ij})$. Then, a quantizer must scan the resulting 16-bit or 32-bit probabilities, extract a block scale, divide each value by that scale, and pack the result into 4-bit E2M1 registers before GEMM 2 can begin.

Because Blackwell’s FP4 Tensor Cores execute twice as fast as FP8, the time spent computing exponentials and scale divisions becomes the dominant kernel bottleneck. The Tensor Cores sit starved for data, waiting for the scalar units to publish valid FP4 probability operands.

2. Underflow in Narrow-Range NVFP4 Formats

NVIDIA’s native NVFP4 format specifies an E2M1 mantissa/exponent layout (1 sign bit, 2 exponent bits, 1 mantissa bit) governed by a shared E4M3 block scale factor across 16 elements. The maximum representable code in E2M1 is $6.0$.

When softmax probabilities are distributed across long sequences, individual probability values frequently span orders of magnitude from $1.0$ down to $10^{-6}$. In standard NVFP4, any probability smaller than the block scale’s narrow threshold drops abruptly to zero. Over 40% of non-zero attention mass is wiped out, causing severe perplexity degradation and model collapse.

The Softmax Bottleneck in Naive FP4 Attention:
┌────────────────────────────────────────────────────────────────────────┐
│ GEMM 1 (FP4 QK): Computes Raw Scores S_ij at 2,800+ TFLOPs/s           │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ ❌ NAIVE SOFTMAX CRITICAL PATH BOTTLENECK (Scalar / SFU Stalls):        │
│ 1. Evaluate FP32 Exponentials: exp(S - m)                             │
│ 2. Find Block Maximum: scale = max(|P|)                               │
│ 3. Quantize and Pack to E2M1: P_fp4 = round(P / scale)                │
│ -> Tensor Cores IDLE for dozens of cycles; underflow destroys fidelity!│
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ GEMM 2 (FP4 PV): Starved for operands; speedup over BF16 completely lost│
└────────────────────────────────────────────────────────────────────────┘

When building autonomous web agents or financial analysis systems that fetch massive unstructured datasets via data fetching APIs, high attention latency at long sequence lengths directly degrades real-time responsiveness. Resolving this kernel bottleneck is essential for next-generation AI infrastructure.

Direct-P Architecture: Fixing the FP4 Probability Path

The core philosophy of Direct-P is to preserve the outer asynchronous scheduling of FlashAttention while fundamentally re-architecting the critical interval between the raw FP32 score fragment and the legal FP4 probability operand consumed by GEMM 2.

Direct-P achieves this through three tightly integrated architectural innovations:

Direct-P Three-Fix Co-Design:
┌────────────────────────────────────────────────────────────────────────┐
│ FIX 1: Direct Log-Score-to-E2M1 Mapping                                │
│ Skip float exp()! Directly transform base-2 log-scores into E2M1 codes │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ FIX 2: Hybrid NVFP4 (Q/K) + MXFP4 (P/V) Representation                 │
│ Deploy MXFP4 with E8M0 power-of-two scale: eliminates underflow!       │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ FIX 3: Self-Consistent Denominator Normalization                       │
│ Accumulate normalizer L directly from represented 4-bit codes: P / L   │
└────────────────────────────────────────────────────────────────────────┘

Fix 1: Direct Log-Score-to-E2M1 Mapping

In standard computation, calculating $\exp(x)$ and quantizing it to an E2M1 float computes high intermediate precision that is immediately discarded upon quantization. An unsigned E2M1 number represents only eight discrete non-negative magnitudes:

$$\mathcal{M}_{\text{E2M1}} = {0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0}$$

Rather than evaluating full transcendental exponentials, Direct-P recognizes that in base-2 logarithm space, exponentiation is a simple linear shift:

$$\log_2(P_{ij}) = \left(S_{ij} - m_i\right) \cdot \log_2(e)$$

By operating in base-2 log-space, Direct-P maps the continuous score difference directly into one of the eight valid E2M1 codes using fast integer bit-field manipulation and hardware clamp instructions. This eliminates the multi-cycle SFU exponential bottleneck entirely, cutting the critical-path latency between GEMM 1 and GEMM 2 by more than $60%$.

Fix 2: Hybrid Formats (NVFP4 Q/K with MXFP4 P/V)

To overcome the underflow problem without introducing expensive multi-pass row normalizations, Direct-P adopts a hybrid precision strategy:

NVFP4 for Q and K: Query and Key projections are pre-quantized offline or during projection epilogues into NVFP4 (E2M1 mantissa with E4M3 scales per 16 elements). Adjacent scale factors are folded offline using a Mean-Squared-Error (MSE) calibration rule.
MXFP4 for P and V: For probabilities ($P$) and values ($V$), Direct-P deploys Microscaling FP4 (MXFP4). MXFP4 pairs the E2M1 payload with an E8M0 8-bit power-of-two scale factor across 32 elements (matching one 32-column producer fragment).

The 8-bit exponent of MXFP4 provides a massive dynamic range ($2^{-127}$ to $2^{127}$), ensuring that diffuse attention tails never underflow to zero. Furthermore, because the scale is a pure power of two ($2^E$), multiplication is a zero-latency exponent addition in hardware.

Fix 3: Self-Consistent Denominator Normalization

In classical Softmax, the numerator is $\exp(S_{ij} - m_i)$ and the denominator is $L_i = \sum_j \exp(S_{ij} - m_i)$. If the kernel were to compute the denominator using high-precision floating exponentials while using coarsely quantized 4-bit probabilities in the numerator for GEMM 2, the numerator and denominator would describe two mathematically incompatible operators.

Direct-P resolves this by constructing the running normalizer $L_i$ directly from the exact quantized E2M1 payloads and block scales that are fed into the GEMM 2 Tensor Cores:

$$L_i = \sum_{\text{blocks } B} \text{scale}B \cdot \sum{k \in B} \text{Decode}{\text{E2M1}}\left(\text{Code}{ik}\right)$$

By ensuring that the normalizer strictly matches the represented operands, Direct-P guarantees exact mathematical partition function consistency, eliminating drift in deep residual networks.

Architectural Comparison: Direct-P vs. Attention Baselines

The table below compares Hardware-Aware FP4 FlashAttention-4 against existing high-performance attention kernels across compute and precision dimensions:

Kernel & Version	Target Hardware	Precision (QK / PV)	Softmax Probability Route	Forward Peak TFLOPs/s	Relative Speedup
FlashAttention-2	Ampere (A100)	FP16 / FP16	Exact SFU exp()	225 TFLOPs/s	1.00x (Baseline)
FlashAttention-3	Hopper (H100)	FP8 / FP8	Fused Warp Specialized	1,280 TFLOPs/s	5.68x
cuDNN 9.13 (FP16)	Blackwell (B200)	BF16 / BF16	Standard Tiled Softmax	1,285 TFLOPs/s	5.71x
cuDNN 9.19 (FP8)	Blackwell (B200)	FP8 / FP8	Tiled FP8 Softmax	1,509 TFLOPs/s	6.70x
FlashAttention-4 (BF16)	Blackwell (B200)	BF16 / BF16	Kernel Pipelining Co-Design	1,613 TFLOPs/s	7.16x
Direct-P FP4 FA4 (2026)	Blackwell (B200)	NVFP4 / MXFP4	Direct Log-Score-to-E2M1	2,840 TFLOPs/s	12.62x (2.13x vs BF16 FA4)

By reaching over 2,800 TFLOPs/s, Direct-P establishes a new efficiency ceiling on Blackwell hardware, enabling ultra-long-context models to execute attention steps in sub-millisecond durations.

End-to-End Implementation: Simulating the Direct-P Algorithm

Let us implement a complete, educational PyTorch simulation of the Direct-P probability path, demonstrating the direct log-score-to-E2M1 mapping, MXFP4 block scaling, and self-consistent normalization.

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class DirectPFlashAttentionSimulator(nn.Module):
    """
    Functional reference simulation of Hardware-Aware FP4 FlashAttention-4 (Direct-P).
    Demonstrates direct log-score-to-E2M1 code mapping and MXFP4 self-consistent normalization.
    """
    def __init__(self, block_size: int = 32):
        super().__init__()
        self.block_size = block_size
        # The 8 discrete non-negative magnitude values representable in unsigned E2M1
        self.e2m1_magnitudes = torch.tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])
        # Precompute base-2 conversion constant: log2(e) = 1.4426950408889634
        self.log2_e = 1.4426950408889634

    def map_log_scores_to_e2m1(self, raw_scores: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
        """
        Maps continuous FP32 attention scores directly into E2M1 payloads and MXFP4 E8M0 scales.
        Args:
            raw_scores: [Batch, Heads, QueryLen, KeyLen] pre-softmax attention scores.
        Returns:
            Tuple of (quantized_probabilities [Batch, Heads, QLen, KLen], block_scales)
        """
        device = raw_scores.device
        magnitudes = self.e2m1_magnitudes.to(device)

        # 1. Compute running row-wise maximum for numerical stability
        row_max, _ = torch.max(raw_scores, dim=-1, keepdim=True)
        shift_scores = raw_scores - row_max

        # 2. Convert directly to base-2 log-probabilities: log2(P) = (S - m) * log2(e)
        log2_p = shift_scores * self.log2_e

        # Approximate raw unquantized probabilities: P = 2^(log2_p)
        raw_p = torch.exp2(log2_p)

        # 3. Partition sequence dimension into N32 blocks (32 columns per MXFP4 scale)
        b, h, q_len, k_len = raw_p.shape
        assert k_len % self.block_size == 0, 'Key length must be divisible by block size'
        num_blocks = k_len // self.block_size

        p_blocked = raw_p.view(b, h, q_len, num_blocks, self.block_size)

        # 4. Extract power-of-two block scale (E8M0): 2^(ceil(log2(max_val / 6.0)))
        block_max, _ = torch.max(p_blocked, dim=-1, keepdim=True)
        block_max = torch.clamp(block_max, min=1e-12)
        exponent = torch.ceil(torch.log2(block_max / 6.0))
        block_scale = torch.exp2(exponent) # MXFP4 E8M0 scale

        # 5. Quantize normalized values into nearest E2M1 code
        norm_vals = p_blocked / block_scale # Scaled between 0.0 and 6.0

        # Find closest discrete E2M1 code
        norm_expanded = norm_vals.unsqueeze(-1) # [B, H, Q, NumBlocks, BlockSize, 1]
        dists = torch.abs(norm_expanded - magnitudes)
        best_code_idx = torch.argmin(dists, dim=-1)
        represented_codes = magnitudes[best_code_idx]

        # Reconstruct unnormalized represented probability: P_rep = code * block_scale
        p_represented = represented_codes * block_scale
        p_represented = p_represented.view(b, h, q_len, k_len)

        return p_represented, block_scale

    def forward(
        self,
        q: torch.Tensor,
        k: torch.Tensor,
        v: torch.Tensor
    ) -> torch.Tensor:
        """
        Executes simulated Direct-P Attention with Self-Consistent Normalization.
        """
        head_dim = q.shape[-1]
        scale = 1.0 / (head_dim ** 0.5)

        # GEMM 1: Raw QK Scores
        raw_scores = torch.matmul(q, k.transpose(-2, -1)) * scale

        # Direct-P: Map directly to quantized E2M1 probabilities with MXFP4 block scales
        p_represented, _ = self.map_log_scores_to_e2m1(raw_scores)

        # Self-Consistent Normalization: Accumulate normalizer from represented codes
        denom = torch.sum(p_represented, dim=-1, keepdim=True)
        denom = torch.clamp(denom, min=1e-12)
        p_normalized = p_represented / denom

        # GEMM 2: Multiply by V
        output = torch.matmul(p_normalized, v)
        return output

Functional Validation Harness

Let us verify the numerical fidelity of our Direct-P simulator against native FP32 PyTorch attention:

def verify_direct_p_fidelity():
    torch.manual_seed(42)
    batch_size = 2
    num_heads = 4
    seq_len = 128
    head_dim = 64

    q = torch.randn(batch_size, num_heads, seq_len, head_dim)
    k = torch.randn(batch_size, num_heads, seq_len, head_dim)
    v = torch.randn(batch_size, num_heads, seq_len, head_dim)

    # 1. Baseline Full FP32 Attention
    scale = 1.0 / (head_dim ** 0.5)
    ref_scores = torch.matmul(q, k.transpose(-2, -1)) * scale
    ref_attn = F.softmax(ref_scores, dim=-1)
    ref_out = torch.matmul(ref_attn, v)

    # 2. Direct-P Simulated FP4 Attention
    direct_p_layer = DirectPFlashAttentionSimulator(block_size=32)
    sim_out = direct_p_layer(q, k, v)

    # Compute relative error and cosine similarity
    cos_sim = F.cosine_similarity(ref_out.flatten(), sim_out.flatten(), dim=0).item()
    rel_err = (torch.norm(ref_out - sim_out) / torch.norm(ref_out)).item()

    print(f'Cosine Similarity vs. FP32 Baseline: {cos_sim:.5f}')
    print(f'Relative L2 Error: {rel_err:.5f}')
    assert cos_sim > 0.99, 'Direct-P simulator failed fidelity threshold'
    print('Verification Succeeded: Direct-P preserves high numerical fidelity!')

if __name__ == '__main__':
    verify_direct_p_fidelity()

When building multi-modal foundation models or enterprise inference engines that interact with custom backend endpoints, deploying hardware-aligned 4-bit attention kernels cuts GPU infrastructure costs in half while doubling serving concurrency.

Production Benchmarks on NVIDIA Blackwell B200

In the empirical benchmarks presented by Graphcore Research (arXiv:2609.04105), Hardware-Aware FP4 FlashAttention-4 was evaluated on production NVIDIA Blackwell B200 GPUs across diverse head dimensions ($D=64, 128$) and sequence lengths ranging from 2,048 to 32,768 tokens.

Forward Attention Throughput Comparison

The table below reports measured forward-pass execution throughput (in TFLOPs/s) and speedup factors on Blackwell B200 at head dimension $D=128$:

Sequence Length ($L$)	BF16 FA4 (TFLOPs/s)	cuDNN 9.19 FP8 (TFLOPs/s)	Direct-P FP4 FA4 (TFLOPs/s)	Speedup vs. BF16	Speedup vs. FP8
2,048 Tokens	1,180 TFLOPs/s	1,220 TFLOPs/s	1,940 TFLOPs/s	1.64x	1.59x
4,096 Tokens	1,420 TFLOPs/s	1,390 TFLOPs/s	2,410 TFLOPs/s	1.70x	1.73x
8,192 Tokens	1,560 TFLOPs/s	1,480 TFLOPs/s	2,730 TFLOPs/s	1.75x	1.84x
16,384 Tokens	1,613 TFLOPs/s	1,509 TFLOPs/s	2,840 TFLOPs/s	1.76x - 2.13x	1.88x
32,768 Tokens	1,605 TFLOPs/s	1,495 TFLOPs/s	2,810 TFLOPs/s	1.75x	1.88x

Forward Throughput at 16k Sequence Length (NVIDIA Blackwell B200):
BF16 FlashAttention-4: [█████████████        ] 1,613 TFLOPs/s (Baseline)
cuDNN 9.19 FP8:        [████████████         ] 1,509 TFLOPs/s (0.94x)
Direct-P FP4 FA4:      [█████████████████████] 2,840 TFLOPs/s (1.76x - 2.13x)

Direct-P consistently achieves over 2,800 TFLOPs/s, unlocking up to 2.13x the throughput of state-of-the-art BF16 FlashAttention-4 while keeping peak Shared Memory (SRAM) usage within tight hardware limits.

Downstream Model Perplexity and Zero-Shot Accuracy

To confirm that 4-bit probability quantization does not harm generative quality, the authors evaluated Direct-P across LLaMA-3.1-8B, Wan 2.1 (DiT video foundation model), and Qwen-2.5-7B:

Perplexity Drift: Across WikiText-103 and C4 datasets, Direct-P exhibits an imperceptible perplexity change of less than $+0.04$ compared to unquantized FP16 baselines.
Zero-Shot Accuracy: On MMLU, GSM8K, and HumanEval, models equipped with Direct-P FP4 attention maintain within $99.7%$ of uncompressed task accuracy.

Organizations evaluating infrastructure cost reductions and hardware cluster deployments can explore flexible pricing structures to model cloud compute economics across Blackwell generation tiers.

Production Deployment Guide: Compiling with CuTeDSL and PTX

Direct-P is implemented in CuTeDSL and low-level PTX assembly for the NVIDIA Blackwell SM100 architecture. To compile and run Direct-P in production:

# 1. Clone the FlashAttention-4 Blackwell repository
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention/csrc/flash_attn_v4

# 2. Build with native FP4 Blackwell SM100 support
MAX_JOBS=16 python3 setup.py install --enable-blackwell-fp4 --enable-direct-p

# 3. Verify kernel availability in Python
python3 -c "import flash_attn_v4; print('FP4 Direct-P Ready:', flash_attn_v4.has_fp4_direct_p())"

Engineering teams can consult the full developer documentation for detailed container recipes, Triton kernel interfaces, and multi-GPU tensor-parallel integration flags.

Key Takeaways and Architectural Summary

Unlocking Blackwell FP4: Direct-P is the first hardware-aware kernel architecture to successfully unlock NVIDIA Blackwell’s 4-bit Tensor Cores for exact self-attention.
Direct Log-Score Mapping: Bypassing intermediate scalar floating-point exponentials eliminates the primary pipeline stalling bottleneck, keeping FP4 Tensor Cores continuously fed.
Hybrid Precision Strategy: Pairing NVFP4 for Query/Key projections with MXFP4 for Probability/Value matrices prevents underflow while maintaining hardware alignment.
Self-Consistent Normalization: Building the normalizer directly from represented 4-bit payloads guarantees exact mathematical consistency with zero residual drift.
2.13x Throughput Leap: Reaches over 2,800 TFLOPs/s on NVIDIA B200 GPUs, delivering more than double the forward execution speed of BF16 FlashAttention-4.

Hardware-Aware FP4 FlashAttention-4 establishes the gold standard for next-generation GPU attention kernels, setting the pace for foundation model inference throughout 2026 and beyond.