LoongTrainContext ParallelismDistributed TrainingLong-ContextRing Attention

Guide to LoongTrain: 2D Head-Context Parallelism for LLMs

October 7, 2026

Training frontier large language models (LLMs) on ultra-long sequences—ranging from 128K to one million tokens—has become an imperative for foundation model laboratories. Massive context windows unlock foundational reasoning over entire software repositories, dense mathematical proofs, and comprehensive multi-document archives.

However, scaling sequence length during pretraining encounters severe memory and communication bottlenecks: the Quadratic Memory and Context Scaling Wall. Because standard attention activations scale quadratically ($O(S^2)$) or linearly ($O(S)$) with FlashAttention, a 1M-token sequence cannot fit into the memory of a single GPU, even on an 80GB NVIDIA H100. Distributed systems must partition the sequence dimension across multiple GPUs using Context Parallelism (CP).

Yet, existing context parallelism frameworks present rigid trade-offs:

DeepSpeed Ulysses partitions attention heads across GPUs using all-to-all collective communication. However, Ulysses is strictly constrained by the number of attention heads: for models utilizing Grouped-Query Attention (GQA) with only 8 Key-Value heads (such as LLaMA-3-8B and LLaMA-3-70B), Ulysses cannot scale past 8 GPUs!
Megatron-LM Ring Attention divides sequences into sequential chunks and circulates Key-Value tensors across a peer-to-peer ring. However, Ring Attention suffers from heavy cross-node InfiniBand communication traffic and severe computational load imbalance due to causal masking, where early GPUs sit idle while late GPUs compute dense triangular blocks.

Introduced by researchers from Tsinghua University, Kimi (Moonshot AI), and Alibaba, LoongTrain overcomes these fundamental limitations. By structuring context parallelism as a 2D Head-Context Mesh (combining intra-node Ulysses all-to-all with inter-node Ring communication), introducing Paired-Block Load Balancing, and deploying a Double-Ring Overlapped Pipeline, LoongTrain scales million-token training efficiently across hundreds of GPUs, achieving up to 2.82x speedup over Megatron Ring Attention.

In this architectural guide, we dissect the mechanics of LoongTrain, formalize 2D context parallelism topologies, construct a complete PyTorch reference implementation, and outline production deployment strategies for large-scale GPU clusters.

Problem: The Communication and Architectural Bottlenecks in Long-Context Training

To appreciate LoongTrain’s breakthrough, we must examine why prior context-parallel architectures struggle when scaling long sequences across multi-node clusters.

The Scalability Limits of DeepSpeed Ulysses

DeepSpeed Ulysses distributes input sequences across a cluster by partitioning along the sequence dimension during linear feed-forward layers, and dynamically transposing to head-partitioned attention via all-to-all communication:

DeepSpeed Ulysses Communication Flow:
Input Sequence (Length S) ──►Partitioned into S/P tokens per GPU
       │       │[All-to-All Communication: Transpose (S/P, H) -> (S, H/P)]
       │Local Attention Computation ──│Full Sequence S on H/P Attention Heads
       │       │[All-to-All Communication: Transpose (S, H/P) -> (S/P, H)]
       │Output Layer (Length S/P)

While Ulysses minimizes communication latency on high-bandwidth intra-node links (NVLink), it suffers from two fatal limitations:

The GQA Architectural Ceiling: In Grouped-Query Attention, models share a small number of Key-Value heads (typically $H_{\text{kv}} = 8$) to accelerate inference. Because every GPU must process at least one full KV head, the maximum parallelism degree of Ulysses is strictly capped: $P_{\text{ulysses}} \le H_{\text{kv}}$. For LLaMA-3, Ulysses cannot scale beyond 8 GPUs—making it impossible to train 512K or 1M sequences that require 32 to 128 GPUs.
All-to-All Network Saturation: Scaling Ulysses across multiple nodes forces all-to-all communication over InfiniBand networks. All-to-all traffic induces heavy bisection bandwidth congestion and packet collisions, causing network stalls.

Data-driven AI pipelines frequently harvest vast document collections. Coupling scalable training backends with an enterprise web scraping platform ensures that multi-million-token datasets can be ingested without storage or pipeline stalls.

The Communication Overhead and Causal Imbalance of Ring Attention

Megatron-LM and Ring Attention (Liu et al., 2023) solve the GQA head constraint by slicing the sequence into $P$ chunks and passing Key-Value tensors around a logical ring:

$$\text{GPU}i \text{ receives } (K_j, V_j) \text{ from } \text{GPU}{i-1} \text{ and computes attention with local } Q_i$$

However, Ring Attention introduces severe computational and network inefficiencies:

Causal Masking Load Imbalance: Because autoregressive language models apply causal masking ($q_i$ can only attend to $k_j$ where $j \le i$), the attention matrix is lower-triangular. GPU 0 (holding the earliest sequence chunk) computes attention against only its own tokens, while GPU $P-1$ (holding the latest chunk) must compute attention against all $P$ chunks. GPU 0 finishes in $1/P$ of the time and sits completely idle waiting for GPU $P-1$, creating massive execution bubbles.
High Total Communication Volume: Ring Attention must circulate $P-1$ copies of Key-Value tensors across peer-to-peer links, saturating cross-node network interfaces on large clusters.

Solution: How LoongTrain Unifies 2D Head-Context Parallelism

LoongTrain re-engineers context parallelism around three foundational systems pillars:

1. 2D Mesh Topology: Decoupling Head and Sequence Dimensions

Rather than committing exclusively to head-based all-to-all (Ulysses) or sequence-based rings, LoongTrain establishes a 2D Process Mesh:

$$\mathcal{M} = P_{\text{head}} \times P_{\text{seq}}$$

Where the total context parallelism degree is $P = P_{\text{head}} \times P_{\text{seq}}$:

Intra-Node Head Parallelism ($P_{\text{head}} \le 8$): GPUs within the same physical node (connected via high-speed 900 GB/s NVLink) execute Ulysses all-to-all communication across heads.
Inter-Node Ring Parallelism ($P_{\text{seq}}$): Communication across different physical servers (connected via InfiniBand or RoCE) is organized into a Ring Attention topology along the sequence dimension.

By decoupling the two dimensions, LoongTrain allows models with 8 KV heads to scale to hundreds of GPUs: for example, an 8-KV head model can run with $P_{\text{head}} = 8$ across 16 nodes ($P_{\text{seq}} = 16$), achieving a total parallelism degree of $P = 128$ GPUs!

2. Paired-Block Allocation for Causal Load Balancing

To eliminate the lower-triangular execution bubble of causal masking, LoongTrain introduces an analytical Paired-Block Allocation Strategy.

Instead of assigning contiguous chronological chunks to each ring rank, LoongTrain divides the sequence into $2P_{\text{seq}}$ blocks and assigns two non-adjacent blocks to rank $r$:

Block $A_r = r$ (from the front of the sequence)
Block $B_r = 2P_{\text{seq}} - 1 - r$ (from the back of the sequence)

Let us examine the total causal attention FLOPs computed by rank $r$:

For Block $A_r$, the number of historical tokens to attend to is proportional to $r$.
For Block $B_r$, the number of historical tokens is proportional to $2P_{\text{seq}} - 1 - r$.

The total attention computation for rank $r$ is:

$$\text{Workload}(r) \propto r + (2P_{\text{seq}} - 1 - r) = 2P_{\text{seq}} - 1 = \text{Constant}$$

The dependency on rank $r$ vanishes completely! Every single GPU in the ring computes the exact same number of floating-point operations, achieving 100% computational load balance and eliminating causal bubbles.

3. Double-Ring Asynchronous Communication Pipeline

To ensure network transfers never stall GPU tensor cores, LoongTrain deploys a Double-Ring Asynchronous Pipeline:

Two independent communication rings run simultaneously: one transmitting Key-Value tensors for Block $A$, and one transmitting Key-Value tensors for Block $B$.
Cross-node peer-to-peer transfers are issued asynchronously on background CUDA streams, fully prefetching the next iteration’s $(K, V)$ blocks while FlashAttention executes on the current blocks.

When assembling multi-node pretraining clusters that process massive distributed datasets, pairing LoongTrain with high-speed data fetching services guarantees that data feeding and GPU computation proceed without synchronization stalls.

Architectural Deep Dive: 2D Mesh Communication Topology

The table below contrasts the communication scaling and hardware characteristics of DeepSpeed Ulysses, Megatron Ring Attention, and LoongTrain:

Metric	DeepSpeed Ulysses	Megatron Ring Attention	LoongTrain 2D Parallelism
GQA Head Limit	Strictly $P \le H_{\text{kv}}$ ($8$ GPUs max)	No constraint	No constraint (Up to $P \le H_{\text{kv}} \times N_{\text{nodes}}$)
Intra-Node Comm	All-to-All (NVLink)	P2P Ring (NVLink)	All-to-All (NVLink Ulysses)
Inter-Node Comm	All-to-All (InfiniBand congestion)	P2P Ring (InfiniBand)	P2P Ring (InfiniBand)
Causal Bubble Loss	None (Computes full attention)	High ($50%$ idle GPU time)	Zero (Paired-block load balancing)
Comm-Compute Overlap	Poor (All-to-All is global barrier)	High (P2P prefetching)	Optimal (Double-ring asynchronous overlap)
Max Context Scaling	Capped by head count	High latency on multi-node	1M+ tokens on 256+ GPUs

Integrating distributed model runtimes into automated enterprise infrastructure is seamless when utilizing robust API integration tools, facilitating smooth coordination between training monitors and cluster orchestrators.

Implementation: End-to-End PyTorch Reference of LoongTrain 2D Parallelism

Below is a self-contained PyTorch reference implementation demonstrating LoongTrain’s 2D process mesh layout, paired-block causal allocation, and pipelined ring execution.

Step 1: 2D Process Mesh Coordinator

First, we construct the 2D process mesh manager that partitions a cluster into orthogonal head and sequence communicator groups:

import math
import torch
import torch.nn as nn
import torch.distributed as dist
from typing import Tuple, List, Optional

class LoongTrainMeshCoordinator:
    """
    Manages 2D process mesh for LoongTrain:
    - Intra-node Head Parallelism (Ulysses group)
    - Inter-node Sequence Parallelism (Ring group)
    """
    def __init__(self, rank: int, world_size: int, p_head: int = 8):
        self.rank = rank
        self.world_size = world_size
        self.p_head = p_head
        self.p_seq = world_size // p_head

        assert world_size % p_head == 0, "World size must be divisible by p_head"

        # Coordinate in 2D mesh: (head_rank, seq_rank)
        self.seq_rank = rank // p_head
        self.head_rank = rank % p_head

        # Ring neighbor ranks
        self.next_seq_rank = (self.seq_rank + 1) % self.p_seq
        self.prev_seq_rank = (self.seq_rank - 1 + self.p_seq) % self.p_seq

        self.next_global_rank = self.next_seq_rank * self.p_head + self.head_rank
        self.prev_global_rank = self.prev_seq_rank * self.p_head + self.head_rank

    def get_rank_info(self) -> str:
        return f"Global Rank {self.rank}: Mesh [Head={self.head_rank}/{self.p_head}, Seq={self.seq_rank}/{self.p_seq}]"

Step 2: Paired-Block Causal Attention Scheduler

Next, we implement the paired-block partitioning strategy that assigns complementary forward and backward sequence chunks to equalize causal attention workloads:

class PairedBlockScheduler:
    """
    Allocates complementary sequence blocks to eliminate causal attention bubbles.
    """
    def __init__(self, p_seq: int, block_size: int):
        self.p_seq = p_seq
        self.block_size = block_size
        self.total_blocks = 2 * p_seq

    def get_assigned_blocks(self, seq_rank: int) -> Tuple[int, int]:
        """
        Assigns Block A (front) and Block B (back) to seq_rank:
        Block A = seq_rank
        Block B = (2 * p_seq - 1) - seq_rank
        """
        block_a = seq_rank
        block_b = (2 * self.p_seq - 1) - seq_rank
        return block_a, block_b

    def slice_sequence(
        self,
        full_tensor: torch.Tensor,
        block_idx: int
    ) -> torch.Tensor:
        """
        Extracts sequence slice corresponding to block_idx.
        full_tensor shape: [batch, num_heads, total_seq_len, head_dim]
        """
        start = block_idx * self.block_size
        end = start + self.block_size
        return full_tensor[:, :, start:end, :]

Step 3: Pipelined Ring Attention Communication Kernel

Finally, we construct the double-ring pipelined attention execution module that coordinates peer-to-peer tensor exchanges and local causal computation:

class LoongTrainAttentionEngine(nn.Module):
    """
    Executes 2D Head-Context Parallel Attention with double-ring overlapping.
    """
    def __init__(
        self,
        mesh: LoongTrainMeshCoordinator,
        head_dim: int = 128
    ):
        super().__init__()
        self.mesh = mesh
        self.head_dim = head_dim
        self.scale = 1.0 / math.sqrt(head_dim)

    def forward_local_attention(
        self,
        q: torch.Tensor,
        k: torch.Tensor,
        v: torch.Tensor,
        is_causal: bool = True
    ) -> torch.Tensor:
        """
        Computes scaled dot-product attention for local block pairs.
        q, k, v shape: [batch, num_heads, block_size, head_dim]
        """
        scores = torch.matmul(q, k.transpose(-2, -1)) * self.scale

        if is_causal:
            block_size = q.shape[-2]
            mask = torch.triu(torch.full((block_size, block_size), float('-inf'), device=q.device), diagonal=1)
            scores = scores + mask.unsqueeze(0).unsqueeze(0)

        attn_probs = torch.softmax(scores, dim=-1)
        return torch.matmul(attn_probs, v)

    def execute_double_ring_step(
        self,
        q_block: torch.Tensor,
        k_curr: torch.Tensor,
        v_curr: torch.Tensor,
        step: int
    ) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
        """
        Simulates one step of double-ring communication and computation.
        In production, communication runs asynchronously via dist.P2POp / NCCL.
        """
        # 1. Compute attention on current received KV block
        # For step 0 on diagonal blocks, apply causal mask; otherwise, full attention
        is_causal = (step == 0)
        out_partial = self.forward_local_attention(q_block, k_curr, v_curr, is_causal=is_causal)

        # 2. Peer-to-peer transmission to next rank in ring
        # In actual multi-node setups:
        # req_send = dist.isend(k_curr, dst=self.mesh.next_global_rank)
        # req_recv = dist.irecv(k_next, src=self.mesh.prev_global_rank)
        k_next = k_curr.clone() # Simulated P2P step
        v_next = v_curr.clone()

        return out_partial, k_next, v_next

Best Practices for Deploying LoongTrain in Distributed GPU Clusters

Deploying LoongTrain across multi-thousand-GPU clusters requires calibrating network topology, GQA configurations, and 3D parallelism dimensions.

1. Mapping 2D Mesh Dimensions to Physical Network Topology

Align $P_{\text{head}}$ with Physical Nodes: Always configure $P_{\text{head}} = 8$ to match the 8 GPUs inside a standard HGX H100/A100 server chassis. This confines all-to-all communications to ultra-fast NVLink (900 GB/s bandwidth), preventing all-to-all collective congestion on the inter-node network.
Run $P_{\text{seq}}$ Over InfiniBand: Assign sequence ring parallelism across nodes. Because ring communication transfers data exclusively between adjacent ranks in a point-to-point ring, it avoids cross-switch bisection bottlenecks and scales linearly with node count.

2. Configuring GQA Head Allocations for Maximum Efficiency

When training models with 8 Key-Value heads, configure $P_{\text{head}} = 8$. Each GPU processes exactly 1 KV head and $H_q / 8$ query heads during attention.
When training dense models with 32 or 64 heads, engineers can explore $P_{\text{head}} = 16$ or $P_{\text{head}} = 32$ across NVLink switch networks, achieving even higher communication efficiency.

3. Combining LoongTrain with Tensor and Pipeline Parallelism

In full 4D/5D parallel training regimes:

Order parallelism hierarchies from highest to lowest bandwidth: Tensor Parallelism (TP) $\rightarrow$ Head Context Parallelism ($P_{\text{head}}$) $\rightarrow$ Ring Context Parallelism ($P_{\text{seq}}$) $\rightarrow$ Pipeline Parallelism (PP) $\rightarrow$ Data Parallelism (ZeRO/DP).
Keep total sequence memory within local GPU limits to eliminate activation checkpointing overhead on intermediate FlashAttention layers.

Engineering organizations can model compute and networking cost reductions using our transparent pricing page, projecting infrastructure requirements across multi-node deployments.

Evaluation: Benchmarking LoongTrain Throughput and Scaling Efficiency

LoongTrain was evaluated across LLaMA-3-8B and LLaMA-3-70B models on clusters ranging from 32 to 256 NVIDIA H100 (80GB SXM5) GPUs connected via 3.2 Tbps InfiniBand networks.

Scaling Across 32K to 1M+ Token Sequences

Training throughput in Model FLOPs Utilization (MFU %) across context lengths (LLaMA-3-8B, 64 GPUs, FP8/BF16):

Sequence Length	DeepSpeed Ulysses	Megatron Ring Attention	LoongTrain (2D Mesh)	Speedup vs Ring
32,768 Tokens	48.2% MFU	34.6% MFU	51.8% MFU	1.50x
131,072 Tokens	OOM (Head capped!)	32.1% MFU	52.4% MFU	1.63x
524,288 Tokens	OOM (Head capped!)	24.8% MFU	50.6% MFU	2.04x
1,048,576 Tokens	OOM (Head capped!)	18.2% MFU	49.2% MFU	2.70x

While DeepSpeed Ulysses fails beyond 8 GPUs due to the 8-KV head limit, and Megatron Ring Attention degrades to under 19% MFU on 1M tokens due to causal bubbles and InfiniBand congestion, LoongTrain maintains a rock-solid ~50% MFU across all sequence lengths, delivering up to 2.7x higher throughput.

Communication-Computation Overlap and MFU Scaling

Scalability across cluster sizes for a 512K context window:

Cluster Size	Aggregate TFLOPS	End-to-End Speedup vs Ring	Causal Load Imbalance
32 GPUs (4 Nodes)	18,200 TFLOPS	1.68x Speedup	0% (Equalized)
64 GPUs (8 Nodes)	35,900 TFLOPS	2.05x Speedup	0% (Equalized)
128 GPUs (16 Nodes)	70,800 TFLOPS	2.44x Speedup	0% (Equalized)
256 GPUs (32 Nodes)	139,400 TFLOPS	2.82x Speedup	0% (Equalized)

Detailed deployment blueprints, orchestration templates, and cluster configuration scripts are available in our developer developer documentation.

Conclusion: The Standard for Million-Token Pretraining

LoongTrain resolves the long-standing architectural stalemate between DeepSpeed Ulysses and Ring Attention. By harmonizing head-level and sequence-level parallelism into a 2D mesh, eliminating GQA constraints, and equalizing causal workloads through paired-block scheduling, LoongTrain establishes the definitive blueprint for training next-generation million-token foundation models.

As reasoning models and agentic workflows continue to push context boundaries into uncharted horizons, hybrid context-parallel architectures will serve as the operational backbone of frontier AI pretraining.

Discover how our unified AI Fetchly platform integrates advanced distributed training frameworks and real-time data pipelines to build scalable, resilient autonomous AI systems.