Weave Guide: Co-Scheduling Disaggregated RL Training

Introduction: The Economic Crisis in Reinforcement Learning Post-Training

Reinforcement Learning (RL) post-training—spanning algorithms such as Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO)—has become the defining methodology for aligning frontier foundation models. Post-training endows language models with multi-step reasoning capabilities, tool-use proficiency, mathematical rigor, and human preference adherence. In enterprise production systems powering automated web scraping swarms, continuous data fetching pipelines, and autonomous API integration copilots, fine-tuning reasoning models via RL post-training is essential for operational reliability.

However, training massive foundation models (from 8B to 70B+ parameters) via reinforcement learning introduces an acute cluster resource dilemma:

1. The Colocation Catastrophe: Traditional RL frameworks (such as Ray-PPO and early DeepSpeed-Chat) colocate the Rollout Phase (generating token completions and sampling rollouts) and the Training Phase (computing policy gradients, backpropagating loss, and updating optimizer states) on the same GPU cluster. Colocating these workloads causes severe memory thrashing: model weights, optimizer states (AdamW FP32 states), and multi-turn KV cache buffers compete for High-Bandwidth Memory (HBM), throttling batch sizes and reducing GPU utilization.
2. The Disaggregation Paradox: To avoid memory interference, modern hyperscale providers decouple infrastructure into specialized clusters: a Rollout Cluster (optimized for memory bandwidth and low-latency decoding, e.g., NVIDIA H20 or L40S) and a Training Cluster (optimized for dense Tensor Core FLOPs and high-speed AllReduce interconnects, e.g., NVIDIA H800 or H100).
3. The 50% Structural Dependency Bubble Tax: While disaggregation eliminates memory contention, on-policy RL algorithms impose strict phase synchronization: training cannot begin until all rollouts are generated, and the next rollout iteration cannot begin until policy weights are updated. Consequently, while the Rollout Cluster is actively generating tokens, the expensive Training Cluster sits completely idle. Conversely, while the Training Cluster computes gradients, the Rollout Cluster sits completely idle.

At the scale of hundreds or thousands of GPUs, this structural lockstep synchronization wastes over 50% of available GPU hours, stranding millions of dollars in datacenter capital.

Enter Weave, a breakthrough cluster scheduling framework presented at the 2026 USENIX Symposium on Operating Systems Design and Implementation (OSDI 2026) by Tianyuan Wu, Lunxi Cao, Yining Wei, Wei Gao, Yuheng Zhao, and collaborators from the Hong Kong University of Science and Technology (HKUST), University of Illinois Urbana-Champaign (UIUC), and Alibaba Cloud.

Weave eliminates the disaggregation tax through an elegant architectural insight: the structural idleness of one RL job can be reclaimed by the active phase of another job. By introducing a Co-Execution Group Abstraction that enforces warm-start host memory residency, an inter-group scheduler utilizing conservative stochastic planning, and an intra-group scheduler orchestrating a provably optimal round-robin dispatch, Weave multiplexes complementary RL jobs across shared rollout and training pools.

Evaluated on a massive production testbed comprising 328 NVIDIA H20 GPUs and 328 NVIDIA H800 GPUs, Weave improves cost efficiency by $1.84\times$ over standard disaggregation and $1.38\times$ over state-of-the-art colocated baselines while achieving $100%$ SLO attainment.

In this technical guide, we dissect the mechanics of disaggregated RL dependency bubbles, analyze the co-execution group abstraction, examine Weave’s two-tier scheduling algorithms, construct a complete Python/PyTorch reference implementation, and explore the architectural implications for enterprise AI training infrastructure in 2026.

---

Anatomy of the Disaggregated RL Dependency Bubble

To understand why traditional cluster managers cannot prevent GPU idling in disaggregated RL, we must analyze the temporal lifecycle of an on-policy reinforcement learning iteration.

The On-Policy RL Iteration Loop

In on-policy reinforcement learning (such as PPO), training proceeds in synchronized cycles:

$$\text{Iteration } k: \quad \mathcal{D}k \sim \pi{\theta_k}(\cdot \mid x) \quad \longrightarrow \quad \theta_{k+1} = \operatorname{Train}(\theta_k, \mathcal{D}_k)$$

This workflow decomposes into two distinct operational phases:

• Phase 1: Rollout Generation ($\mathcal{T}_{\text{rollout}}$): The current policy $\pi_{\theta_k}$ generates $N$ candidate trajectories from prompt dataset $\mathcal{X}$. Generates tokens autoregressively, evaluates rewards via external verifiers or reward models, and computes Generalized Advantage Estimation (GAE).
• Phase 2: Policy Training ($\mathcal{T}_{\text{train}}$): The policy trainer consumes collected trajectories $\mathcal{D}k$, computes surrogate actor-critic losses, executes backpropagation across multiple micro-batches, and performs AdamW optimizer updates to produce $\theta{k+1}$.

Why Existing Cluster Schedulers Fail to Reclaim the Bubble

When datacenter operators attempt to solve this idling problem using standard cluster schedulers (such as Slurm, Kubernetes, or Ray), three systemic barriers emerge:

1. The Prohibitive Cost of Cold-Start Context Switching

The naive approach to filling an idle bubble is time-slicing: when Job A finishes its training phase and begins rollouts, the training cluster could theoretically run Job B.

However, foundation models maintain enormous memory state. A 70B parameter model requires:

• Model weights (FP16): $\sim 140\text{ GB}$
• AdamW optimizer states (FP32 master weights, momentum, variance): $\sim 560\text{ GB}$
• Gradients and activation buffers: $\sim 100\text{ GB}$

Reloading hundreds of gigabytes from remote storage (Ceph, Lustre, or NVMe-oF) across the network requires 30 to 90 seconds. If the idle bubble is 60 seconds long, cold-start model loading consumes the entire bubble, introducing thrashing and net negative throughput.

2. Micro-Batch Runtime Skew and Variance

Rollout duration is non-deterministic. Depending on prompt complexity, completion length, and tool execution latency, the rollout duration $\mathcal{T}_{\text{rollout}}$ fluctuates dynamically across iterations.

Standard static schedulers cannot adapt to dynamic phase duration jitter. If Job B’s training phase overruns its scheduled window by 15 seconds, it blocks Job A from starting its training phase, triggering cascading deadline violations and breaking on-policy synchronization.

3. Asymmetric Hardware Resource Ratios

Different RL jobs exhibit disparate computational ratios. A math reasoning model (e.g., DeepSeek-Math) may require long, multi-turn reasoning rollouts ($\mathcal{T}{\text{rollout}} \gg \mathcal{T}{\text{train}}$), whereas a dense multimodal alignment job may require heavy gradient updates over large image embeddings ($\mathcal{T}{\text{train}} \approx \mathcal{T}{\text{rollout}}$). Arbitrarily pairing jobs without considering phase duration ratios causes severe structural mismatch.

---

The Weave Architecture: Principles and Core Mechanics

Weave resolves this cluster efficiency crisis by introducing cross-cluster co-scheduling. Rather than treating disaggregated clusters as isolated hardware silos, Weave manages Rollout Clusters and Training Clusters as a co-optimized cooperative system.

Weave is built upon three foundational architectural pillars:

1. The Co-Execution Group Abstraction: Partitions the global cluster into isolated locality domains and enforces a warm-start residency constraint, ensuring model parameters remain permanently cached in local host DRAM to enable sub-50ms context switching.
2. The Inter-Group Scheduler: Macro-level cluster planning that groups complementary RL jobs based on conservative stochastic optimization, pairing jobs whose phase duration ratios naturally interlock.
3. The Intra-Group Scheduler: Micro-level runtime dispatcher that orchestrates a provably optimal round-robin schedule across paired jobs, dynamically adapting to runtime variance without violating training deadlines.

---

Deep Dive: Co-Execution Groups & Two-Tier Scheduling

Let us examine the operational mechanisms that allow Weave to multiplex multi-model training without performance interference.

1. The Co-Execution Group Abstraction & Warm-Start Residency

To eliminate multi-minute model reloading delays, Weave introduces the Co-Execution Group Abstraction.

A Co-Execution Group $\mathcal{G}$ consists of a set of co-scheduled jobs $\mathcal{J}{\mathcal{G}} = {J_1, J_2, \dots, J_m}$ mapped to a dedicated subset of Rollout GPUs $\mathcal{R}{\mathcal{G}}$ and Training GPUs $\mathcal{T}_{\mathcal{G}}$.

Weave enforces a strict Host Memory Residency Constraint: $$\sum_{J_i \in \mathcal{J}{\mathcal{G}}} \text{Memory}{\text{Host}}(J_i) \le \text{Capacity}_{\text{Node_DRAM}}$$

Because modern GPU server nodes (such as NVIDIA HGX H100/H800) feature $1.5\text{ TB}$ to $2.0\text{ TB}$ of high-speed DDR5 host system memory, the server can comfortably hold the weights and optimizer states of multiple 32B or 70B models in pinned host RAM simultaneously.

When the intra-group scheduler switches the training cluster from Job A to Job B:

1. Job A’s GPU-resident optimizer states are yielded via high-speed PCIe Gen5 DMA directly into host DDR5 memory.
2. Job B’s cached parameters are immediately streamed from pinned host RAM into GPU HBM.

Because the data never touches remote storage arrays or network sockets, the entire context switch completes in under $50\text{ milliseconds}$.

2. The Inter-Group Scheduler: Stochastic Job Pairing

How does Weave decide which jobs to pair together into a co-execution group?

For each job $J_i$, let $\mathcal{T}{\text{rollout}}(J_i)$ and $\mathcal{T}{\text{train}}(J_i)$ represent its empirical phase durations. Weave models these durations as random variables with known means $\mu$ and variances $\sigma^2$ derived from historical iterations.

The Ideal Pairing Condition for a two-job group ${J_A, J_B}$ occurs when the rollout time of Job A matches the training time of Job B, and vice-versa:

$$\mathcal{T}{\text{rollout}}(J_A) \approx \mathcal{T}{\text{train}}(J_B) \quad \text{and} \quad \mathcal{T}{\text{rollout}}(J_B) \approx \mathcal{T}{\text{train}}(J_A)$$

When this condition holds, the training cluster is 100% saturated executing Job B while Job A rolls out, and executes Job A while Job B rolls out.

To handle non-stationary variance and avoid deadline violations, the Inter-Group Scheduler solves a Conservative Stochastic Optimization Problem. It seeks job pairings that minimize aggregate idle bubble expectation while bounding the probability of an SLO overrun:

$$\min_{{\mathcal{G}_1, \dots, \mathcal{G}K}} \sum{k=1}^K \mathbb{E}\left[ \text{Bubble}(\mathcal{G}_k) \right] \quad \text{subject to} \quad \mathbb{P}\left( \text{Latency}(\mathcal{G}_k) > \text{SLO} \right) \le \epsilon$$

Using branch-and-bound integer programming, Weave partitions cluster jobs into optimal co-execution groups in under 500 milliseconds.

3. The Intra-Group Scheduler: Dynamic Round-Robin Dispatch

At runtime within each group, the Intra-Group Scheduler enforces an interleaved execution flow:

If Job A’s rollout finishes slightly earlier than expected, the scheduler dynamically queues lightweight data pre-processing or checkpoint saving for Job B to absorb the slack, maintaining perfect synchronization without stalls.

---

Python & PyTorch Implementation: Building a Weave Co-Scheduler

To demonstrate how Weave organizes co-execution groups, enforces warm-start host memory residency, and interleaves phase execution, let us construct a fully functional simulation in Python and PyTorch.

Our implementation includes:

1. RLJob: Models on-policy RL jobs with model size, phase duration distributions, and memory footprints.
2. WarmStartMemoryManager: Simulates pinned host memory caching and sub-50ms context switching.
3. WeaveCoExecutionGroup: Orchestrates two-job interleaved round-robin scheduling across disaggregated pools.
4. WeaveClusterSimulator: Evaluates cluster utilization and cost efficiency comparing standard disaggregation against Weave.

---

Architectural Comparison: Weave vs. RL Scheduling Baselines

To contextualize Weave’s contributions, the following table compares Weave against existing RL post-training cluster management architectures across core systems dimensions:

Systems Dimension	Colocated RL (Ray-PPO / OpenRLHF)	Standard Disaggregated RL	Slurm Static Time-Slicing	Weave (OSDI 2026)
Cluster Topology	Unified GPU cluster (Trainer+Rollout)	Disaggregated Rollout & Train pools	Disaggregated Pools	Disaggregated Co-Execution Groups
GPU Memory Thrashing	High (Weights + KV + Opt states compete)	None (Hardware separated)	None (Hardware separated)	None (Hardware separated)
Dependency Bubbles	None (Single job sequential)	Severe (50%+ GPU idle time)	Severe (Clusters idle between jobs)	Reclaimed via Cross-Job Multiplexing
Context Switch Latency	Not applicable	Not applicable	30s–90s (Remote storage reload)	$\< 50\text{ ms}$ (Pinned Host DRAM DMA)
Job Scheduling Metric	First-Come-First-Served	Independent FIFO queues	Coarse Priority Queue	Two-Tier Stochastic Planning
Cluster Hardware Scale	Moderate (128–512 GPUs)	Large (512–1,024 GPUs)	Large	656 Production GPUs (Alibaba Cloud)
Cost Efficiency Gain	1.0x (Baseline)	0.85x–1.10x (High idle cost)	0.95x	1.84x over Disaggregation (1.38x vs Colocated)

While colocated training avoids cross-cluster dependency bubbles, it forces batch sizes to shrink to fit memory, crippling rollout throughput. Standard disaggregation fixes memory interference but wastes over half of cluster time in idle bubbles. Weave uniquely captures the best of both worlds: dedicated hardware efficiency with zero idle bubble waste.

---

Empirical Production Results & Industrial Evaluation

In the USENIX OSDI 2026 proceedings, Wu et al. evaluated Weave across extensive real-world post-training workloads on an Alibaba Cloud production testbed comprising 656 GPUs (328 NVIDIA H20 inference accelerators and 328 NVIDIA H800 training GPUs):

1. Cost Efficiency Scaling

Evaluated across diverse post-training benchmarks—including mathematical reasoning (LLaMA-3-8B/70B), code generation (Qwen-2.5-Coder), and multi-turn instruction following:

• Compared against standard disaggregated RL architectures, Weave achieved a $1.84\times$ improvement in cluster cost efficiency (measured as completed training iterations per dollar).
• Compared against state-of-the-art colocated baselines (which suffer from memory-throttled batch sizes), Weave delivered a $1.38\times$ cost efficiency gain.

2. Elimination of Structural Idling Bubbles

The authors monitored hardware utilization across 48 hours of continuous multi-job execution:

• Under standard disaggregation, average GPU compute core utilization hovered at $46.2%$, with over $53%$ of time spent in dependency waiting states.
• Under Weave’s interleaved co-scheduling, aggregate GPU compute utilization climbed to $89.4%$, successfully reclaiming over $82%$ of theoretical structural bubbles.

3. Warm-Start Context Switch Verification

Weave’s host memory residency constraint reduced context-switch latency from $52.4\text{ seconds}$ (remote storage reload) down to $48\text{ milliseconds}$, completely eliminating context-switch overhead from the critical training path.

---

Step-by-Step Implementation Guide: Deploying Weave Principles

For ML infrastructure engineers, cluster platform architects, and reinforcement learning research teams looking to adopt Weave’s co-scheduling principles, follow this implementation roadmap:

Step 1: Separate Rollout and Training Infrastructure Pools

Partition your Kubernetes or Slurm cluster into dedicated hardware tiers:

• Rollout Tier: Deploy high-memory-bandwidth, cost-effective inference accelerators (e.g., NVIDIA L40S, A100, H20).
• Training Tier: Deploy high-TFLOPS, NVLink-dense training nodes (e.g., NVIDIA H100 SXM, H800).

Step 2: Implement Pinned Host Memory Residency

Configure your model serving and training daemons to retain inactive model weights in pinned host DRAM rather than evicting them to disk:

Step 3: Configure Two-Job Interleaved Orchestration

In your cluster orchestrator (e.g., custom Ray controller or Kubernetes operator), pair complementary RL jobs and interleave their phase execution:

For enterprises building automated high-throughput scraping pipelines with our data fetching infrastructure or fine-tuning enterprise reasoning copilots, eliminating RL cluster idling translates directly into faster model iteration cycles and lower cloud compute costs. To explore custom enterprise infrastructure hosting, visit our pricing page.

---

Future Outlook: The Evolution of Cloud-Scale Post-Training

The publication of Weave at USENIX OSDI 2026 establishes cross-cluster multiplexing as an essential architectural primitive in modern AI datacenters. As reinforcement learning post-training expands from small language models to trillion-parameter multimodal reasoning agents, static cluster allocation is no longer viable.

Looking toward 2027 and beyond, we anticipate several vital developments:

1. Heterogeneous Three-Tier Co-Scheduling: Extending Weave’s co-execution abstractions to three-tier pipelines: co-scheduling rollout, environmental tool execution, and policy training across separate specialized hardware pools.
2. Elastic Spot-Instance Post-Training: Dynamically migrating rollout phases to low-cost ephemeral spot instances worldwide while keeping training centralized on high-bandwidth supercomputing clusters.
3. Hardware-Triggered Context Migration: Future server architectures integrating direct CXL (Compute Express Link) 3.0 fabrics to achieve sub-millisecond context migrations between host memory and accelerator HBM.

For more technical architecture breakdowns, systems tutorials, and deployment guides, visit our documentation hub or learn how AIFetchly accelerates AI engineering teams across the world.

---

Conclusion & Strategic Takeaways

As reinforcement learning post-training becomes the dominant driver of foundation model intelligence, eliminating infrastructure idling is critical for economic viability.

The strategic takeaways from Weave include:

• Disaggregation Creates Structural Bubbles: Separating rollout and training eliminates memory thrashing, but on-policy synchronization leaves each cluster idle for over 50% of the time.
• Reclaim Bubbles Across Jobs: The structural idleness of one job can be utilized by the active phase of another, doubling effective hardware throughput.
• Warm-Start Residency is Mandatory: Storing inactive model parameters in pinned host DRAM reduces context-switch latency from 50 seconds to under 50 milliseconds.
• 1.84x Cost Efficiency at Production Scale: Evaluated on 656 production GPUs at Alibaba Cloud, Weave demonstrates that cross-cluster co-scheduling is the definitive blueprint for high-efficiency RL post-training.

---