FastVideo Guide: Sliding Tile Attention AI Model

Problem: The 3D Attention Bottleneck in Video Diffusion Transformers

Diffusion Transformers (DiTs)—exemplified by frontier architectures like HunyuanVideo, CogVideoX, and Wan2.1—have established an undisputed dominance in high-definition video synthesis. By replacing legacy convolutional U-Nets with scalable transformer backbones, modern video generation models synthesize cinematic footage featuring fluid physical motion, photorealistic lighting dynamics, and intricate narrative consistency. Whether rendering synthesized video campaigns from web assets harvested via automated media scraping pipelines or generating interactive visual streams from real-time streaming data ingestion, video DiTs represent the frontier of generative media.

However, behind these breathtaking visual capabilities lies an excruciating computational barrier: the quadratic attention bottleneck.

The 115K Token Sequence Wall

In static image generation, processing a $512 \times 512$ image requires approximately $1,024$ patch tokens. Even with dense quadratic self-attention, evaluating a single image requires modest computational resources.

In video generation, however, models must capture spatial dimensions ($H \times W$) across time ($T$). After 3D VAE latent compression, generating a standard 5-second 720P video clip ($1280 \times 768$ at 24 FPS, corresponding to 117 raw frames) yields a latent tensor of shape $(30, 48, 80)$. When flattened into visual tokens, the resulting sequence length explodes to:

$$N = 30 \times 48 \times 80 = 115,200 \text{ tokens!}$$

Calculating full bidirectional 3D attention over $115,200$ tokens requires an attention matrix of:

$$N^2 = (115,200)^2 \approx 1.327 \times 10^{10} \text{ elements per attention head per layer!}$$

Across 40 transformer layers and 50 denoising timesteps, attention alone accounts for over 800 seconds out of the 945 seconds of total generation time on an NVIDIA H100 SXM5 GPU. Generating just 5 seconds of video takes nearly 16 minutes on a $30,000 accelerator!

The Inefficiency of Traditional Sliding Window Attention (SWA)

Profiling pretrained video diffusion models reveals an intuitive physical phenomenon: 3D Spatio-Temporal Locality. Video signals exhibit extreme temporal and spatial correlation: adjacent pixels and neighboring frames carry the vast majority of mutual information. In HunyuanVideo, empirical analysis demonstrates that a local 3D window covering only 15.52% of the token space accounts for over 70% of total attention recall.

This finding suggests that Sliding Window Attention (SWA) should drastically accelerate video DiTs by restricting attention to localized 3D cuboids.

However, in practice, existing 2D and 3D sliding window kernels (such as NATTEN, Tiled NATTEN, and CLEAR) fail completely on GPUs:

1. The Mixed-Block Trap: High-performance attention kernels (like FlashAttention) divide queries and keys into hardware-friendly matrix blocks (e.g., $64 \times 64$ or $128 \times 128$). In traditional SWA, each query token slides its window independently. This creates a staggered, zigzag boundary across attention blocks, producing numerous mixed blocks (blocks that are partially masked and partially dense).
2. Extreme Mask Evaluation Overhead: A mixed block cannot simply be skipped. The GPU must calculate the exact geometric distance between every query-key pair on chip to apply dynamic masking. In 3D space, this mask computation introduces enormous overhead, causing kernels like NATTEN to run slower than unmasked dense FlashAttention despite executing 90% fewer theoretical FLOPs!
3. Catastrophic Hardware Underutilization: While dense FlashAttention-3 achieves over 64% Model Flops Utilization (MFU), 3D NATTEN and CLEAR collapse to an abysmal 3.7% to 5.4% MFU.

To solve this hardware-algorithm disconnect and achieve real wall-clock speedups matching theoretical sparsity, researchers at the Hao AI Lab at UC San Diego, in collaboration with UC Berkeley, Tsinghua University, and MBZUAI, developed FastVideo: Fast Video Generation with Sliding Tile Attention (arXiv:2502.04507).

---

Solution: Sliding Tile Attention (STA) and System-Algorithm Co-Design

FastVideo introduces Sliding Tile Attention (STA), a hardware-aware attention paradigm that replaces token-by-token sliding with tile-by-tile sliding.

By shifting the sliding unit from an individual token to a hardware-native tile, FastVideo completely eliminates mixed blocks: every single attention block in STA is either 100% dense or 100% empty.

This allows the attention kernel to achieve 58.79% MFU—accelerating the attention operator by up to 10.45x over FlashAttention-3 and cutting end-to-end HunyuanVideo generation latency from 945 seconds down to 268 seconds.

To explore how these high-throughput inference engines integrate into scalable enterprise agent architectures, visit AI Fetchly.

---

Architecture: Mathematical Foundations of Sliding Tile Attention

Sliding Tile Attention is built on the principle that hardware efficiency requires regularized memory access. Rather than allowing every individual token to carve out an idiosyncratic neighborhood, tokens are grouped into compact 3D spatio-temporal tiles.

Mathematical Formulation and Mixed Block Elimination

Let a video latent volume have spatial-temporal dimensions $(L, L, L)$, containing $N = L^3$ tokens. Let the FlashAttention SRAM block size be $B$.

In 3D Sliding Tile Attention:

1. We define the tile dimension $T$ such that: $$B = T^3$$ For example, if FlashAttention operates on block size $B = 64$, we define $T = 4$ (since $4^3 = 64$).
2. The entire video sequence is partitioned into non-overlapping 3D tiles of shape $(T, T, T)$. Tokens within each tile are indexed consecutively in memory.
3. The sliding window size $W$ is chosen to be an exact integer multiple of $T$.
4. A query tile at coordinate $(i, j, k)$ slides over key tiles within a cubic window of radius $W$: $$\mathcal{N}(i, j, k) = \left{ (i’, j’, k’) ;\Big|; |i - i’| \le \frac{W}{2}, ; |j - j’| \le \frac{W}{2}, ; |k - k’| \le \frac{W}{2} \right}$$

Because all queries inside tile $(i, j, k)$ share the exact same neighborhood $\mathcal{N}(i, j, k)$:

• The attention sub-matrix between any query tile and any key tile is either 100% full (dense) or 100% zero (empty).
• The number of mixed blocks is mathematically zero:

$$N_{\text{mix}}^{\text{STA}} = 0$$

$$N_{\text{dense}}^{\text{STA}} = \left(\frac{W}{T}\right)^3 \cdot \left(\frac{L}{T}\right)^3$$

In contrast, traditional 3D Tiled NATTEN produces a massive proportion of mixed blocks:

$$N_{\text{mix}}^{\text{NATTEN}} = \left(2 \left\lceil \frac{W - 1}{2T} \right\rceil + 1\right)^3 \cdot \left(\frac{L}{T}\right)^3 - N_{\text{dense}}^{\text{NATTEN}}$$

The ThunderKittens Producer-Consumer Architecture

To maximize hardware utilization, FastVideo implements STA using ThunderKittens, a fast embedded domain-specific language for NVIDIA Hopper (H100) Tensor Cores:

• Disaggregated Mask Logic: In standard attention, the compute kernel must evaluate complex coordinate distances for every thread. In STA, the sparse attention pattern is managed entirely by producer warpgroups.
• Asynchronous KV Prefetching: While the consumer warpgroups compute Tensor Core matrix multiplications on tile $k$ in high-speed SRAM, the producer warpgroups asynchronously load key and value tiles $k+1$ from High-Bandwidth Memory (HBM) using Tensor Memory Accelerator (TMA) instructions.
• Zero Compute Masking Overhead: Consumer warpgroups are completely oblivious to the sparsity pattern. They execute standard, unmasked, dense GEMMs over the circular SRAM cache, achieving 94.09% of dense FlashAttention kernel efficiency.

Head Specialization and Automated Window Profiling

Not all attention heads require the same window size:

• Local Detail Heads: Focus on sharp physical textures (hair, water droplets, edge contrast) and require only small 3D windows (e.g., $w = (18, 24, 24)$).
• Global Scene Heads: Model holistic background motion, lighting transitions, and long-range semantic interactions, requiring broader windows (e.g., $w = (30, 40, 40)$).

FastVideo introduces an automated, training-free profiling algorithm:

1. Profiles attention recall across a small calibration set of 16 text prompts.
2. Evaluates the Mean Squared Error (MSE) between dense attention output and candidate STA window patterns for each head $(l, h)$.
3. Assigns an optimal per-head window configuration that preserves over 99.8% visual fidelity while cutting total attention FLOPs by up to 75%.

---

Implementation: PyTorch Sliding Tile Attention and Kernel Mechanics

Below is a complete, production-grade PyTorch implementation demonstrating the 3D tile partitioning, tile-level sliding window mapping, and dense block execution of Sliding Tile Attention:

For enterprises orchestrating distributed GPU clusters with automated video processing workflows, review our architectural guide on streaming API integrations.

---

Comparison: Quantitative and Wall-Clock Latency Benchmarks

FastVideo and Sliding Tile Attention were benchmarked on HunyuanVideo (generating 5-second 720P clips, 117 frames, 115,200 tokens) on NVIDIA H100 SXM5 GPUs, evaluated against FlashAttention-2 (FA2), FlashAttention-3 (FA3), CLEAR, NATTEN, Tiled NATTEN, and Swin:

Key Benchmark Discoveries

1. Converts Theoretical Sparsity into Real Hardware Speedups: While NATTEN and CLEAR slow down execution due to irregular masking overhead, FastVideo’s Sliding Tile Attention accelerates the attention kernel by 10.45x over FlashAttention-3 and 17x over FlashAttention-2.

2. Drastic End-to-End Generation Acceleration (268s vs. 945s): FastVideo slashes HunyuanVideo 720P generation time from 16 minutes down to 4.4 minutes (a 3.53x speedup). In training-free mode, latency drops to 501 seconds with zero fine-tuning.

3. VBench Benchmark Parity (82.62% vs. 82.71%): On the comprehensive VBench suite (assessing subject consistency, background consistency, temporal flickering, and motion smoothness), FastVideo preserves full generation quality with a statistically negligible 0.09% variance.

---

Production: Enterprise Deployment, Fine-Tuning Recipes, and Serving

Deploying FastVideo in enterprise production environments—such as cloud media rendering engines, generative video APIs, and game asset pipelines—requires optimized deployment strategies:

The FastVideo Fine-Tuning Recipe

To achieve the maximum 3.53x speedup at 91% attention sparsity, fine-tune the video DiT using FastVideo’s tri-loss objective:

1. Hardware Configuration: Requires 8x NVIDIA H100 GPUs for only 8 hours of training (costing under $250 in compute), a trivial fraction of foundation pre-training costs.
2. Tri-Loss Formulation: Combine flow matching data loss $\mathcal{L}{\text{data}}$ with intermediate attention distillation loss $\mathcal{L}{\text{attn}}$ and final layer loss $\mathcal{L}{\text{final}}$: $$\mathcal{L}{\text{total}} = \alpha \mathcal{L}{\text{data}} + \beta \mathcal{L}{\text{final}} + \gamma \mathcal{L}_{\text{attn}}$$ with $\alpha = 1.0, \beta = 0.5, \gamma = 0.5$. This teaches intermediate sparse layers to reproduce the global representation of the dense teacher model.

Production Best Practices

1. Keep Initial Timesteps Dense: In diffusion models, the initial 10% to 20% of denoising steps establish macroscopic scene layout and global lighting. Keep full attention for $t \in [T_0, T]$ and activate Sliding Tile Attention for the remaining 80% of steps. This guarantees perfect scene coherence.
2. Match Tile Size to Hardware SRAM: When deploying on NVIDIA Hopper (H100), configure tile size to $(T=4, H=4, W=4)$ to match the 64-token warp execution boundaries. On NVIDIA Blackwell (B200), expand tile size to $(4, 8, 8)$ to saturate enlarged shared memory banks.
3. Cross-Modality Application (Image Super-Resolution): FastVideo’s STA extends directly to 2D image diffusion models (such as FLUX.1). Applying STA to 4K super-resolution upscaling delivers a 3.4x speedup over dense attention without edge blurring.

---

Strategic Summary and Architectural Trajectory

Hao AI Lab’s FastVideo and Sliding Tile Attention demonstrate that solving the generative video compute crisis does not require sacrificing full 3D spatial-temporal modeling.

By diagnosing the fundamental flaw of token-level sliding windows—replacing irregular, mask-heavy operations with clean, hardware-aligned 3D tiles—FastVideo achieves true wall-clock speedups proportional to mathematical sparsity. Delivering a 10.45x acceleration in attention kernels and slashing end-to-end video generation times by up to 3.53x, FastVideo establishes the foundational blueprint for scalable, real-time video diffusion transformers.