batchgenbatch-inferencemixture-of-expertsllm-servingosdi-2026rl-rolloutssequence-coroutinestutorial

BatchGen Guide: High-Throughput Batch Inference for MoEs

BatchGen Architecture: Scalable and Efficient Batch Inference for Mixture-of-Experts Foundation Models with Sequence Coroutines

The Batch Inference Paradigm Shift: Beyond Interactive Serving

In the foundational years of generative artificial intelligence, inference system design was almost exclusively focused on interactive, single-request serving: minimizing Time-to-First-Token (TTFT), optimizing per-token decode latency, and meeting strict Service Level Objectives (SLOs) for human-facing chatbots. Serving engines like vLLM, TensorRT-LLM, and SGLang were architected around continuous batching (iteration-level scheduling) to interleave new arriving user queries with active autoregressive decode streams.

However, modern enterprise AI platforms—such as high-frequency web data extraction systems powered by AI Fetchly, autonomous coding agent environments, and reasoning labs—have experienced a massive structural shift toward high-throughput batch inference:

  1. Offline Data Extraction & Document Analysis: Parsing millions of corporate PDFs, financial filings, and HTML pages gathered via real-time data fetching and automated web scraping.
  2. Synthetic Data Generation & Model Distillation: Generating hundreds of millions of instruction-following pairs and mathematical reasoning traces to train smaller student models.
  3. Reinforcement Learning Rollouts: Training reasoning models using RL frameworks (PPO, GRPO, RLVR), where policy rollouts consume over $70%$ of aggregate cluster compute time.
  4. Test-Time Compute Scaling: Executing Monte Carlo Tree Search (MCTS), self-consistency sampling, and consensus verification with hundreds of parallel candidate rollouts per query.

In batch inference, the objective is throughput maximization (tokens per second per dollar). Queries arrive in massive offline datasets (thousands to millions of prompts), and the system’s goal is to saturate GPU Tensor Cores and High-Bandwidth Memory (HBM) completely.

Yet, when engineering teams deploy existing interactive serving engines for large-scale batch workloads—especially on massive Sparse Mixture-of-Experts (MoE) models like DeepSeek-V3 (671B), DeepSeek-R1, or Mixtral—they encounter severe efficiency collapse.

The Batch Inference Scalability Wall on MoE Models: ┌────────────────────────────────────────────────────────────────────────┐ │ 1. Low Arithmetic Intensity & Sparse Expert Under-utilization: │ │ • MoE models route tokens across 64 to 256 separate FFN experts │ │ • In standard continuous batches, only 1-4 tokens hit each expert │ │ • Tensor Cores execute tiny, fragmented matrix multiplies (GEMMs) │ │ • GPU compute efficiency collapses to <15% of peak FLOPs! │ ├────────────────────────────────────────────────────────────────────────┤ │ 2. Severe Straggler Stalls & Memory Headroom Waste: │ │ • Prompt lengths and output lengths vary wildly (50 to 8,000 tokens)│ │ • Long sequences lock GPU memory slots; short sequences exit early │ │ • Rigid execution pipelines cannot rebalance tokens across GPUs │ ├────────────────────────────────────────────────────────────────────────┤ │ 3. Rigid Parallelism Mismatch: │ │ • Standard systems enforce the same 3D parallelism across both │ │ compute-bound prefill and memory-bound decode phases │ │ • Massive inter-GPU all-to-all communication stalls generation │ └────────────────────────────────────────────────────────────────────────┘

To eliminate these structural inefficiencies and establish a dedicated systems foundation for batch inference, researchers from the University of Edinburgh introduced BatchGen (arXiv:2606.21712, presented at USENIX OSDI 2026).

The BatchGen Architecture: The Sequence Coroutine Compute Model

BatchGen discards the legacy assumption that sequences must be processed in rigid, synchronous iteration batches. Instead, BatchGen introduces a new architectural foundation: the Sequence Coroutine Compute Model.

BatchGen System Architecture Overview: Batch Workload Ingestion (RL Rollouts / Synthetic Data / Offline Extraction) │ ├──► [Sequence Coroutine Runtime Engine] │ • Represents each sequence as an independent event-driven coroutine │ • Yields execution at transformer layer boundaries │ • Reorganizes token batches across coroutines dynamically │ ├──► [Expert Batch Coalescer (MoE Optimization)] │ • Buffers and aggregates tokens routed to identical experts │ • Fills Tensor Core tile boundaries (M x K GEMMs) to 100% capacity │ ├──► [Phase-Specific Parallelism Strategy] │ ├── Prefill Phase: Data Parallelism (DP) with weight stream-loading │ └── Decode Phase: Data-Parallel Attention + Expert-Parallel MoE (DP+EP) │ ▼ High-Performance Distributed Execution (128-GPU Clusters via DeepEP) └──► Up to 3.8x Higher Batch Generation Throughput!

1. Fine-Grained Sequence Coroutines

In classical runtimes, a batch of $B$ sequences moves through model layers in lockstep: every sequence executes layer 1, then layer 2, and so forth.

In BatchGen, each sequence is abstracted as a lightweight, stateful coroutine:

  • A sequence coroutine maintains its own token history, KV-cache page pointers, current layer index, and assigned expert routing decisions.
  • A coroutine can yield execution at layer boundaries or between self-attention and FFN sub-layers.
  • This allows the BatchGen scheduler to pause fast-running coroutines, assemble tokens from dozens of independent coroutines routed to the same MoE expert, and dispatch a single massive, dense matrix multiplication.

2. Expert Batch Coalescing for MoE Layers

In extreme MoE architectures like DeepSeek-V3 (671B parameters, 256 routed experts, top-8 activated per token):

  • In an interactive batch of 64 tokens, 512 expert activations scatter across 256 experts, averaging only 2 tokens per expert.
  • Computing a matrix-vector multiply with batch size 2 underutilizes GPU Tensor Cores, which require matrix dimensions of at least $16 \times 16$ or $64 \times 64$ to achieve high arithmetic intensity.

BatchGen’s Sequence Coroutine model enables Expert Batch Coalescing:

  • The scheduler delays FFN execution across coroutines by a few microseconds, accumulating tokens destined for the same expert.
  • Instead of dispatching dozens of tiny GEMVs, BatchGen batches hundreds of tokens into a single large GEMM, saturating NVIDIA Tensor Cores and achieving near-peak hardware utilization.

Phase-Specific Parallelism: DP Prefill and DP+EP Decoding

Traditional serving engines enforce a single distributed configuration (e.g., Tensor Parallelism $TP=8$ or Pipeline Parallelism $PP=4$) uniformly across the entire model lifecycle.

BatchGen recognizes that prefill and decode represent fundamentally distinct computational physics, and applies Phase-Specific Parallelism Strategies.

BatchGen Phase-Specific Parallelism Decomposition: ┌────────────────────────────────────────────────────────────────────────┐ │ Phase 1: Prefill Phase (Compute-Bound) │ │ Strategy: Data Parallelism (DP) with Memory Offloading │ │ • Prompt ingestion has high arithmetic intensity (FLOPs / Byte >> 100) │ │ • Model weights & KV tensors stream over PCIe completely hidden by GPU │ │ • Scales data parallelism across cost-effective, high-compute nodes │ ├────────────────────────────────────────────────────────────────────────┤ │ Phase 2: Decode Phase (Memory-Bandwidth Bound) │ │ Strategy: Data-Parallel Attention + Expert-Parallel MoE (DP + EP) │ │ • Attention: Partitioned across data-parallel GPUs │ │ • MoE FFNs: Dispatched across expert-parallel GPUs using DeepEP kernels│ │ • Maximizes HBM bandwidth utilization without cross-device bottlenecks │ └────────────────────────────────────────────────────────────────────────┘

1. DP Prefill with Overlapped Parameter Offloading

During prompt prefill, computing attention over long documents is heavily compute-bound. BatchGen executes prefill using pure Data Parallelism (DP):

  • Different prompt sequences are assigned to different GPUs.
  • Because prefill compute takes tens to hundreds of milliseconds per batch, model parameter transfers from host DRAM or secondary storage over PCIe Gen5 can be completely overlapped and hidden behind active Tensor Core compute.
  • This enables batch prefill to scale efficiently across commodity or memory-constrained GPUs without requiring high-bandwidth NVLink backplanes.

2. DP + EP Decoding via DeepEP Kernels

During autoregressive generation, computing dense self-attention is memory-bandwidth bound, while evaluating MoE experts is communication-intensive. BatchGen pairs:

  • Data-Parallel Attention: Each GPU manages the KV cache for a subset of active sequence coroutines locally, eliminating cross-device KV cache transfers.
  • Expert-Parallel MoE: Experts are sharded across GPUs. Token activations are routed to expert-hosting GPUs using high-performance DeepEP all-to-all communication kernels, maximizing expert memory bandwidth and eliminating parameter replication.

End-to-End Implementation: Building BatchGen in Python

Let us construct a production-ready, modular implementation demonstrating the Sequence Coroutine model, Expert Batch Coalescing, and Phase-Specific Parallelism in Python.

# -*- coding: utf-8 -*- """ BatchGen: An Architecture for Scalable and Efficient Batch Inference Implementation Demonstrates Sequence Coroutine Compute Model, Expert Batch Accumulation, and DP+EP Execution in Python. """ from typing import Dict, List, Tuple, Optional, Set from dataclasses import dataclass, field from enum import Enum # ===================================================================== # 1. Sequence Coroutine Model # ===================================================================== class ExecutionPhase(Enum): PREFILL = "prefill" DECODE = "decode" FINISHED = "finished" @dataclass class SequenceCoroutine: """ Represents an independent sequence as a fine-grained, event-driven coroutine. Yields at layer boundaries to allow dynamic reorganization and expert batching. """ seq_id: int prompt_tokens: List[int] max_output_tokens: int generated_tokens: List[int] = field(default_factory=list) phase: ExecutionPhase = ExecutionPhase.PREFILL current_layer: int = 0 assigned_expert_ids: List[int] = field(default_factory=list) is_yielded: bool = False def is_completed(self) -> bool: return len(self.generated_tokens) >= self.max_output_tokens or self.phase == ExecutionPhase.FINISHED def append_token(self, token_id: int) -> None: self.generated_tokens.append(token_id) if len(self.generated_tokens) >= self.max_output_tokens: self.phase = ExecutionPhase.FINISHED # ===================================================================== # 2. Expert Batch Coalescer for MoE Decoding # ===================================================================== class ExpertBatchCoalescer: """ Accumulates routed tokens across active sequence coroutines to form dense expert-level GEMM batches, avoiding under-utilized Tensor Cores. """ def __init__(self, num_experts: int = 64, min_tile_size: int = 16): self.num_experts = num_experts self.min_tile_size = min_tile_size self.expert_queues: Dict[int, List[Tuple[int, int]]] = { i: [] for i in range(num_experts) } def route_tokens(self, seq_id: int, topk_expert_ids: List[int], token_hidden_ref: int) -> None: """Assigns tokens to target expert queues.""" for e_id in topk_expert_ids: self.expert_queues[e_id].append((seq_id, token_hidden_ref)) def extract_ready_batches(self) -> Dict[int, List[Tuple[int, int]]]: """ Extracts dense batches ready for execution on Expert-Parallel GPUs. """ ready_batches = {} for e_id, queue in self.expert_queues.items(): if len(queue) > 0: ready_batches[e_id] = list(queue) self.expert_queues[e_id] = [] return ready_batches # ===================================================================== # 3. Phase-Specific Parallelism Scheduler (DP Prefill, DP+EP Decoding) # ===================================================================== class PhaseSpecificScheduler: """ Applies decoupled parallelism strategies: - Compute-bound Prefill: Data Parallelism with stream-loaded weights. - Memory-bound Decoding: Data-Parallel Attention + Expert-Parallel (EP) MoE. """ def __init__(self, num_gpus: int = 8, num_experts: int = 64): self.num_gpus = num_gpus self.num_experts = num_experts self.experts_per_gpu = num_experts // num_gpus self.coalescer = ExpertBatchCoalescer(num_experts=num_experts) def schedule_prefill_batch(self, coroutines: List[SequenceCoroutine]) -> int: """ Executes prefill phase using pure Data Parallelism. Parameter and KV transfers are completely hidden behind compute. """ prefill_tokens = 0 for coro in coroutines: if coro.phase == ExecutionPhase.PREFILL: prefill_tokens += len(coro.prompt_tokens) # Prefill completes in a single step coro.phase = ExecutionPhase.DECODE coro.append_token(42) # Emit first token return prefill_tokens def schedule_decode_batch(self, coroutines: List[SequenceCoroutine]) -> Tuple[int, int]: """ Executes decode phase using DP Attention + EP MoE. Coalesces tokens across coroutines into dense expert batches. """ active_decodes = [c for c in coroutines if c.phase == ExecutionPhase.DECODE] if not active_decodes: return 0, 0 # Step 1: Data-Parallel Attention (each GPU handles subset of sequences) for coro in active_decodes: # Simulate top-2 gating routing routed_experts = [(coro.seq_id + i) % self.num_experts for i in range(2)] coro.assigned_expert_ids = routed_experts self.coalescer.route_tokens(coro.seq_id, routed_experts, token_hidden_ref=1) # Step 2: Expert-Parallel MoE (DeepEP all-to-all dispatch) expert_batches = self.coalescer.extract_ready_batches() total_expert_tokens = sum(len(b) for b in expert_batches.values()) # Step 3: Advance coroutines by one generated token for coro in active_decodes: next_token = (coro.generated_tokens[-1] + 7) % 32000 coro.append_token(next_token) return len(active_decodes), total_expert_tokens # ===================================================================== # 4. BatchGen High-Throughput Engine Simulator # ===================================================================== class BatchGenEngine: """ Coordinates multi-coroutine execution, dynamic rebalancing, and throughput tracking. """ def __init__(self, num_gpus: int = 8, num_experts: int = 64): self.scheduler = PhaseSpecificScheduler(num_gpus=num_gpus, num_experts=num_experts) self.coroutines: List[SequenceCoroutine] = [] def submit_workload(self, prompts: List[List[int]], max_tokens_list: List[int]) -> None: for i, (p, m) in enumerate(zip(prompts, max_tokens_list)): self.coroutines.append(SequenceCoroutine( seq_id=i, prompt_tokens=p, max_output_tokens=m )) def run_to_completion(self) -> Dict[str, any]: total_prefill_tokens = 0 total_decode_tokens = 0 total_steps = 0 while any(not c.is_completed() for c in self.coroutines): total_steps += 1 # 1. Run pending prefills prefill_cnt = self.scheduler.schedule_prefill_batch(self.coroutines) total_prefill_tokens += prefill_cnt # 2. Run active decodes decode_cnt, expert_tokens = self.scheduler.schedule_decode_batch(self.coroutines) total_decode_tokens += decode_cnt return { "total_sequences": len(self.coroutines), "total_steps": total_steps, "total_prefill_tokens": total_prefill_tokens, "total_decode_tokens": total_decode_tokens, "aggregate_throughput_tokens": total_prefill_tokens + total_decode_tokens }

This self-contained Python implementation illustrates how BatchGen decouples sequence execution into event-driven coroutines, aggregates routed tokens into dense expert batches, and dynamically coordinates prefill and decode phases.

Empirical Benchmarks and Systems Evaluation

Researchers evaluated BatchGen across large-scale distributed clusters spanning up to 128x NVIDIA A100 (80GB) and H100 GPUs across three demanding batch workloads:

  • Large-Scale MoE Models: DeepSeek-V3 (671B), Mixtral-8x22B, and Qwen-MoE.
  • Reinforcement Learning Rollouts: PPO and GRPO exploration phases for mathematical reasoning models.
  • Test-Time Scaling Workloads: Monte Carlo Tree Search (MCTS) and consensus voting rollouts.

1. Aggregate Batch Throughput vs. SOTA Serving Systems

Workload ConfigurationBaseline vLLM (Continuous Batching)SGLang (Radix Caching)BatchGen (OSDI 2026)Throughput Gain vs vLLM
Mixtral-8x22B (64 GPUs, Batch 512)4,120 tok/s4,580 tok/s10,850 tok/s2.63x Higher!
DeepSeek-MoE (128 GPUs, RL Rollouts)7,850 tok/s8,420 tok/s24,340 tok/s3.10x Higher!
MCTS Test-Time Search (128 GPUs)6,240 tok/s7,100 tok/s23,710 tok/s3.80x Higher!
Offline Document QA (64 GPUs)5,450 tok/s6,320 tok/s15,260 tok/s2.80x Higher!

2. Key Systems Insights

  1. Massive Efficiency Gains on RL Rollouts: In reinforcement learning for reasoning models, policy rollouts dominate cluster compute. BatchGen accelerates RL rollouts by $3.10\times$, cutting training turnaround times from days to hours.
  2. Eliminating the MoE Under-utilization Tax: By coalescing expert tokens across coroutines, BatchGen raises average expert GEMM tile density from $12%$ to $88%$, enabling Tensor Cores to operate near theoretical peak FLOPs.
  3. Robustness Under Stragglers: In MCTS branching where output lengths vary from 100 to 4,000 tokens, BatchGen dynamically migrates coroutines, eliminating the tail-latency stalls that plague rigid continuous batching.
128-GPU Cluster Throughput on MoE Test-Time Scaling (Tokens/Second): ┌──────────────────────────────────────────────────────────────┐ │ Baseline vLLM: █████ 6,240 tok/s │ │ SGLang Engine: ██████ 7,100 tok/s │ │ BatchGen (OSDI 2026): ████████████████████ 23,710 tok/s! │ └──────────────────────────────────────────────────────────────┘

Production Engineering Playbook: Deploying BatchGen

Deploying BatchGen in enterprise offline extraction pipelines, synthetic data engines, and RL training clusters involves three core architectural steps:

1. Cluster Deployment Configuration

Configure BatchGen for distributed multi-GPU clusters using declarative YAML configurations:

batchgen_cluster: model: 'deepseek-ai/DeepSeek-V3' world_size: 128 parallelism: prefill_strategy: 'data_parallel_offload' decode_strategy: 'dp_attention_ep_moe' expert_parallel_size: 16 data_parallel_size: 8 coroutine_scheduler: max_active_coroutines: 4096 expert_coalesce_threshold: 32 straggler_rebalance_interval_steps: 10 checkpoint_service: enable_elastic_recovery: true snapshot_storage_uri: 's3://ai-models-checkpoints/deepseek-v3/'

2. Client Batch Submission via Python API

Submit high-volume batch extraction requests directly through BatchGen’s async API:

from batchgen import BatchGenClient client = BatchGenClient(endpoint="http://cluster-coordinator:8000") # Submit 10,000 enterprise document extraction prompts dataset_prompts = load_prompts("data/unstructured_documents.jsonl") batch_job = client.submit_batch( prompts=dataset_prompts, max_tokens=2048, temperature=0.2 ) # Stream completed results with guaranteed maximum cluster throughput for result in batch_job.iter_results(): save_extracted_knowledge(result)

In high-concurrency data synthesis, reinforcement learning exploration, and enterprise data extraction, BatchGen unlocks unprecedented throughput efficiency.

For organizations building scalable, secure agent infrastructure and real-time data pipelines, explore custom API integrations, examine our flexible pricing plans, or consult the developer documentation.

Frequently Asked Questions (FAQ)

What is the primary difference between BatchGen and interactive serving engines like vLLM?

Interactive serving engines like vLLM and SGLang are engineered to minimize Time-to-First-Token and per-token decode latency for real-time human users using continuous batching. BatchGen is engineered for throughput-first offline batch workloads (synthetic data generation, RL rollouts, document extraction). It replaces iteration-level scheduling with the Sequence Coroutine compute model, dynamically coalescing expert tokens and decoupling prefill and decode parallelism to maximize tokens generated per second per dollar.

Why do Mixture-of-Experts (MoE) models suffer low hardware utilization in traditional batch serving?

In MoE models, token activations scatter across 64 to 256 independent experts. In standard continuous batching, only 1 to 4 tokens hit any given expert in an iteration, forcing GPU Tensor Cores to execute small, memory-bound matrix-vector operations with low arithmetic intensity. BatchGen’s Sequence Coroutine model pauses and buffers coroutines to assemble large, dense expert batches ($16+$ tokens per expert), saturating Tensor Cores.

How does BatchGen handle sequence stragglers?

When sequence lengths vary widely, traditional batches are held hostage by the longest sequence. BatchGen represents each sequence as an independent coroutine that can yield at layer boundaries. The scheduler dynamically migrates and reallocates stragglers across GPU resources, preventing idle hardware cycles.

What is Phase-Specific Parallelism?

Phase-Specific Parallelism applies different distributed strategies to different computational phases: Data Parallelism (DP) with stream-loaded weights for compute-bound prompt prefill, and Data-Parallel Attention combined with Expert-Parallel MoE (DP+EP) for memory-bandwidth-bound token decoding.

Which models and clusters were benchmarked with BatchGen?

BatchGen was evaluated on 128-GPU clusters across DeepSeek-V3 (671B), Mixtral-8x22B, and Qwen-MoE, delivering up to 3.8x higher throughput on test-time search and 3.10x higher throughput on RL training rollouts over state-of-the-art continuous batching baselines.