tokenflowllm-servingtext-streamingpreemptive-schedulingkv-cacheeurosys-2026tutorial

TokenFlow Guide: Preemptive LLM Text Streaming Serving

TokenFlow Architecture: Preemptive Request Scheduling and Proactive KV Cache Management for LLM Text Streaming

The Streaming Mismatch: Why Conventional LLM Serving Chokes on Request Bursts

Interactive conversational AI, autonomous agentic programming loops, and live data ingestion platforms engineered with aiFetchly rely fundamentally on streamed token generation. Instead of waiting for a multi-thousand-token sequence to complete execution before returning a monolithic response, modern serving engines progressively transmit tokens across Server-Sent Events (SSE) or WebSockets as they are decoded iteration-by-iteration.

In interactive user-facing streaming, user satisfaction is governed by two orthogonal Service-Level Objectives (SLOs):

  1. Responsiveness (Time-To-First-Token / TTFT): The initial latency before the very first word appears on the user’s screen. High TTFT ($>1.5\text{ s}$) makes applications feel sluggish and unresponsive.
  2. Smoothness (Time-Between-Tokens / TBT): The inter-token generation cadence during ongoing streaming. While humans typically read at a rate of 5 to 8 tokens per second ($125 - 200\text{ ms}$ per token), any intermittent stall exceeding $250\text{ ms}$ is immediately perceived as visual stutter or freezing.

Despite widespread deployment of advanced continuous batching engines like vLLM, TensorRT-LLM, and SGLang, serving clusters suffer a systemic collapse during unanticipated request bursts.

The Non-Preemptive Head-of-Line Blocking Crisis: ┌────────────────────────────────────────────────────────────────────────┐ │ Current Running Batch (Long-Generation Requests 1, 2, 3...) │ │ GPU HBM VRAM: 98% Allocated to active Key-Value (KV) Cache blocks │ └───────────────────────────────────┬────────────────────────────────────┘ │ 💥 High-Priority Burst Arrives: Requests 4, 5, 6 (Short Queries) │ ┌────────────────────┴────────────────────┐ ▼ ▼ Standard Non-Preemptive Engine TokenFlow Preemptive Engine ❌ New requests queued behind running jobs ✅ Checks client buffer occupancy ❌ Wait 4,500ms for running jobs to finish ✅ Preempts streams with surplus buffer ❌ P99 TTFT surges by 5x-10x ✅ Runs new prefill immediately ❌ Wasteful over-generation ahead of human eye ✅ P99 TTFT drops by 80.2%

The Flaws of Non-Preemptive Scheduling and Reactive Memory Management

The inability of existing serving engines to maintain responsiveness under burst traffic stems from two foundational architectural flaws:

1. The Non-Preemptive Run-to-Completion Trap
Existing continuous batching engines operate under a strictly non-preemptive paradigm: once a request enters the running batch, it retains its GPU memory allocation and execution slot until it generates an End-of-Sequence (EOS) token or reaches its maximum token budget. When a burst of new requests arrives, the scheduler cannot schedule their prefill phases because the GPU’s High-Bandwidth Memory (HBM) is fully committed to active Key-Value (KV) caches. New requests sit idle in the waiting queue for seconds while running requests generate hundreds of tokens, causing P99 TTFT to explode.

2. The Generation vs. Consumption Velocity Disconnect
A modern server-class GPU (such as an NVIDIA H100 or H200) executing low-concurrency decode iterations generates tokens at 40 to 100+ tokens per second per request. However, human users and conversational voice agents consume text at only 5 to 8 tokens per second.
As a consequence, the engine spends precious GPU compute cycles and memory bandwidth filling a client-side network buffer with dozens of tokens that the user will not read for several seconds. Meanwhile, new requests that have not even received their first token are completely blocked.

3. Reactive, Synchronous KV Cache Swapping
When engines attempt to free memory by swapping KV caches to host CPU RAM, they do so reactively—only triggering swaps at the exact moment VRAM is exhausted. Because PCIe transfer latency is non-trivial and executed synchronously, the entire GPU pipeline stalls, introducing severe jitter into the ongoing streaming cadence.

To resolve this dilemma, systems researchers from Shanghai Jiao Tong University, George Mason University, and China Telecom introduced TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling (arXiv:2510.02758 / EuroSys 2026). TokenFlow harmonizes generation speed with consumption velocity, preempting ahead-of-time streams and proactively managing KV memory to maximize effective throughput.

Architectural Deep Dive: How TokenFlow Operates

TokenFlow shifts LLM streaming serving from reactive run-to-completion batching to buffer-aware preemptive scheduling coupled with proactive hierarchical memory paging.

┌─────────────────────────────────────────────────────────────────────────────┐ │ TokenFlow Serving Architecture │ │ │ │ Client Connections (SSE / WebSockets) │ │ ▲ │ │ │ [Token Consumption Feedback: C_i(t)] │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ Client Buffer Tracking Manager │ │ │ │ Calculates Real-Time Buffer Slack: S_i(t) = B_i(t) / R_i │ │ │ └──────────────────────────────────┬──────────────────────────────────┘ │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ Dynamic Buffer-Occupancy Priority Scheduler │ │ │ │ • Starving streams -> Highest Priority (Immediate GPU Decode) │ │ │ │ • New requests -> High Priority (Prompt Prefill Interleaving) │ │ │ │ • Surplus streams -> Preempted to CPU (Buffer covers client read) │ │ │ └───────────────────┬───────────────────────────────┬─────────────────┘ │ │ │ │ │ │ ▼ ▼ │ │ ┌───────────────────────────────┐ ┌─────────────────────────────────┐ │ │ │ Proactive KV Cache Pager │ │ GPU Continuous Batching Engine │ │ │ │ Overlapped Async PCIe DMA │ │ Fused Multi-Head Attention │ │ │ │ Pinned CPU Host Memory Pool │ │ High-Throughput Token Generation│ │ │ └───────────────────────────────┘ └─────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘

The system operates across three synergistic pillars:

1. Dynamic Buffer-Occupancy Priority Scheduling

TokenFlow monitors the real-time token buffer state for every active client connection. Let:

  • $G_i(t)$ be the total cumulative tokens generated by the engine for request $r_i$ up to time $t$.
  • $C_i(t)$ be the cumulative tokens consumed (read or rendered) by the client up to time $t$.
  • $B_i(t) = G_i(t) - C_i(t)$ be the active client buffer occupancy.
  • $R_i$ be the client’s measured consumption rate (tokens per second).

TokenFlow defines the Client Buffer Slack Time $S_i(t)$:

$$S_i(t) = \frac{B_i(t)}{R_i}$$

The slack time $S_i(t)$ represents the exact duration the client can continue rendering text smoothly without receiving any new tokens from the server.

When a burst of new requests arrives, TokenFlow does not force them to wait. Instead, it inspects the slack times of currently executing requests:

  • If a running request has $S_i(t) > S_{\text{threshold}}$ (e.g., more than $2.0\text{ seconds}$ of unrendered text buffered in client memory), TokenFlow preempts it.
  • The newly arrived request is allocated GPU memory and executed immediately, completing its prefill phase and returning its first token in tens of milliseconds.
  • Because the preempted request had ample buffer, the client never notices a pause: reading continues seamlessly from the local buffer while the new request is serviced.

2. Proactive Hierarchical KV Cache Paging

Preemption is only viable if the cost of suspending and resuming a request does not destroy GPU compute efficiency. In standard systems, recomputing an evicted prompt wastes enormous compute, while naive swapping causes PCIe transfer stalls.

TokenFlow introduces a proactive, page-level memory manager:

  1. Asynchronous Anticipatory Paging: TokenFlow uses an exponential moving average (EMA) of arrival rates to anticipate preemption. Before an active request is suspended, the scheduler begins asynchronously transferring its oldest KV cache pages from GPU High-Bandwidth Memory (HBM) to pinned host CPU memory via non-blocking PCIe DMA transfers.
  2. Compute-I/O Overlapping: The memory transfers execute concurrently on an asynchronous CUDA stream while the GPU Tensor Cores are actively executing attention matrix multiplications for the remaining batch. By the time a preemption decision is finalized, the KV pages are already in host RAM, incurring zero synchronous blocking penalty.

3. Effective Throughput Maximization

Conventional serving benchmarks evaluate Raw Token Throughput (total tokens generated per second). However, during request bursts, generating tokens at $100\text{ tok/s}$ for an active user who only reads at $6\text{ tok/s}$ while starving a dozen other users yields high raw throughput but abysmal real-world utility.

TokenFlow defines Effective Goodput: the rate of tokens delivered to users strictly within their acceptable interactive deadlines ($TBT \le 200\text{ ms}$ and $TTFT \le 500\text{ ms}$). By dynamically pacing generation to client consumption capacity, TokenFlow unlocks up to 82.5% higher effective goodput across the serving cluster.

Architectural Comparison: TokenFlow vs Traditional Serving Schedulers

Serving CharacteristicFirst-Come-First-Served (vLLM / TGI)Chunked-Prefill Batching (Sarathi)Disaggregated Prefill (DistServe)TokenFlow (EuroSys 2026)
Scheduling PolicyNon-preemptive run-to-completionNon-preemptive iteration chunkingDisaggregated pool assignmentBuffer-aware preemptive scheduling
Burst ResponseHigh queue wait times ($TTFT > 5\text{s}$)Mitigated prefill bubbles, high waitRequires dedicated prefill nodesImmediate preemption (<100ms TTFT)
Memory StrategyReactive swapping on OOMStatic block allocationNetwork RDMA KV transferProactive overlapped async paging
P99 TTFT under BurstHigh ($3,000 - 8,000\text{ ms}$)Moderate ($1,500 - 3,500\text{ ms}$)Good ($800 - 1,200\text{ ms}$)Ultra-Low ($350 - 600\text{ ms}$)
User Reading AwarenessNone (Blind over-generation)NoneNoneReal-time client buffer tracking
Effective GoodputBaseline (1.0x)1.15x1.35x1.82x (+82.5%)
Hardware OverheadZeroMinimalHeavy (Dual GPU clusters + RDMA)Zero (Software-defined scheduler)

Mathematical Formulation of Buffer-Aware Preemption

Let us formalize the scheduling optimization problem governing TokenFlow’s real-time preemption engine.

Client Starvation Hazard Function

Let $\mathcal{A}(t)$ denote the set of active streaming requests at time $t$. For each request $r_i \in \mathcal{A}(t)$, the client buffer contains $B_i(t)$ tokens, consumed at rate $R_i$. The remaining time until starvation is:

$$S_i(t) = \frac{B_i(t)}{R_i}$$

Let $\Delta t$ be the estimated time required to schedule, prefill, and return the first token for an incoming burst request $r_{\text{new}}$:

$$\Delta t_{\text{prefill}} = \tau_{\text{prefill}}\left(L_{\text{prompt}}\right) + \tau_{\text{overhead}}$$

A running request $r_i$ is eligible for preemption if and only if its buffer slack strictly exceeds the preemption window plus a safety margin $\epsilon$:

$$\text{Eligible}(r_i) \iff S_i(t) \ge \Delta t_{\text{prefill}} + \epsilon$$

The Priority Weighting Function

TokenFlow ranks all active and queued requests in a dynamic priority queue using a composite objective function:

$$P_i(t) = -\ln\left(\max\left(S_i(t), ; \delta\right)\right) + \lambda \cdot \frac{1}{\text{WaitTime}_i(t) + \delta} + \mu \cdot \mathbb{I}\left(r_i \text{ is unstarted}\right)$$

Where:

  • $-\ln(S_i(t))$ exponentially boosts the priority of requests as their client buffers approach zero, guaranteeing that ongoing streams never stutter.
  • $\mu \cdot \mathbb{I}(r_i \text{ is unstarted})$ provides a targeted prioritization bonus for newly arrived requests to minimize Time-To-First-Token.
  • $\lambda$ balances fairness across competing requests.

When GPU VRAM capacity is constrained:

$$\sum_{i \in \text{Running}(t)} \text{Pages}(r_i) \le M_{\text{GPU}}$$

TokenFlow selects the subset of requests that maximizes total system priority $\sum P_i(t)$, evicting the lowest-priority requests whose buffer slack is largest.

Step-by-Step Implementation: Building a TokenFlow Serving Engine

Below is a complete, self-contained Python implementation demonstrating TokenFlow’s core components: the Client Buffer Tracker, the Proactive Paged KV Manager with async transfers, and the Preemptive Stream Scheduler.

1. Data Models and System Definitions

""" tokenflow_serving_engine.py Reference Implementation of TokenFlow Preemptive LLM Text Streaming Serving. EuroSys 2026 Reference Architecture. """ from dataclasses import dataclass, field from enum import Enum from typing import List, Dict, Optional, Set import time import math import heapq class StreamState(Enum): QUEUED = "QUEUED" PREFILLING = "PREFILLING" STREAMING = "STREAMING" PREEMPTED = "PREEMPTED" COMPLETED = "COMPLETED" @dataclass class StreamingRequest: request_id: str arrival_time: float prompt_tokens: int max_output_tokens: int client_read_rate: float # tokens per second (typically 5 - 8 tok/s) # Dynamic runtime tracking state: StreamState = StreamState.QUEUED generated_tokens: int = 0 consumed_tokens: int = 0 first_token_time: Optional[float] = None allocated_gpu_pages: List[int] = field(default_factory=list) evicted_cpu_pages: List[int] = field(default_factory=list) @property def buffer_occupancy(self) -> int: return max(0, self.generated_tokens - self.consumed_tokens) @property def slack_seconds(self) -> float: if self.client_read_rate <= 0: return 999.0 return self.buffer_occupancy / self.client_read_rate

2. Proactive KV Cache Pager (GPU / CPU Memory Management)

class PagedKVCacheManager: """ Manages physical KV cache pages across GPU HBM and Pinned Host CPU RAM. Simulates asynchronous background DMA swapping. """ def __init__(self, total_gpu_pages: int = 256, total_cpu_pages: int = 1024, page_size: int = 16): self.total_gpu_pages = total_gpu_pages self.total_cpu_pages = total_cpu_pages self.page_size = page_size self.free_gpu_pages: Set[int] = set(range(total_gpu_pages)) self.free_cpu_pages: Set[int] = set(range(total_cpu_pages)) self.transfer_queue: List[str] = [] def allocate_gpu_pages(self, num_pages: int) -> Optional[List[int]]: if len(self.free_gpu_pages) < num_pages: return None allocated = [self.free_gpu_pages.pop() for _ in range(num_pages)] return allocated def swap_out_to_cpu(self, request: StreamingRequest) -> bool: """Asynchronously evicts request's GPU pages to host CPU memory.""" num_pages = len(request.allocated_gpu_pages) if len(self.free_cpu_pages) < num_pages: return False # CPU memory exhausted # Transfer pages cpu_alloc = [self.free_cpu_pages.pop() for _ in range(num_pages)] request.evicted_cpu_pages.extend(cpu_alloc) # Free GPU pages for page in request.allocated_gpu_pages: self.free_gpu_pages.add(page) request.allocated_gpu_pages.clear() request.state = StreamState.PREEMPTED return True def swap_in_to_gpu(self, request: StreamingRequest) -> bool: """Restores request's KV cache pages from CPU back to GPU VRAM.""" num_pages = len(request.evicted_cpu_pages) if len(self.free_gpu_pages) < num_pages: return False gpu_alloc = [self.free_gpu_pages.pop() for _ in range(num_pages)] request.allocated_gpu_pages.extend(gpu_alloc) for page in request.evicted_cpu_pages: self.free_cpu_pages.add(page) request.evicted_cpu_pages.clear() request.state = StreamState.STREAMING return True

3. The TokenFlow Preemptive Scheduler

class TokenFlowScheduler: """ Dynamic buffer-occupancy priority scheduler with preemptive streaming support. """ def __init__(self, memory_manager: PagedKVCacheManager, min_slack_for_preemption: float = 1.5): self.mem = memory_manager self.min_slack = min_slack_for_preemption self.active_streams: Dict[str, StreamingRequest] = {} self.waiting_queue: List[StreamingRequest] = [] def add_request(self, req: StreamingRequest): self.waiting_queue.append(req) def update_client_consumption(self, current_time: float): """Simulates clients consuming tokens from their local buffers over elapsed time.""" for req in list(self.active_streams.values()): if req.state in [StreamState.STREAMING, StreamState.PREEMPTED]: # Increment consumed tokens based on client reading speed consumed = int((current_time - req.arrival_time) * req.client_read_rate) req.consumed_tokens = min(req.generated_tokens, consumed) def schedule_iteration(self, current_time: float) -> List[StreamingRequest]: """ Executes one scheduling tick: 1. Checks waiting queue for new high-priority prompts. 2. Preempts streams with high buffer slack if memory is exhausted. 3. Returns batch of requests to execute in this iteration. """ self.update_client_consumption(current_time) execution_batch: List[StreamingRequest] = [] # Step 1: Check if new waiting requests need immediate TTFT service while self.waiting_queue: next_req = self.waiting_queue[0] needed_pages = math.ceil(next_req.prompt_tokens / self.mem.page_size) if len(self.mem.free_gpu_pages) >= needed_pages: # Direct allocation self.waiting_queue.pop(0) pages = self.mem.allocate_gpu_pages(needed_pages) next_req.allocated_gpu_pages = pages next_req.state = StreamState.PREFILLING self.active_streams[next_req.request_id] = next_req execution_batch.append(next_req) else: # VRAM full: Look for active streaming requests with surplus buffer slack preempted = self._attempt_preemption(needed_pages) if preempted: # Retry allocation now that pages were freed self.waiting_queue.pop(0) pages = self.mem.allocate_gpu_pages(needed_pages) next_req.allocated_gpu_pages = pages next_req.state = StreamState.PREFILLING self.active_streams[next_req.request_id] = next_req execution_batch.append(next_req) else: # Cannot preempt safely without causing client stutter; wait for completion break # Step 2: Add ongoing active streaming requests to execution batch for req in self.active_streams.values(): if req.state == StreamState.STREAMING: execution_batch.append(req) return execution_batch def _attempt_preemption(self, needed_pages: int) -> bool: """Finds running streams with maximum buffer slack and preempts them to CPU.""" candidates = [ req for req in self.active_streams.values() if req.state == StreamState.STREAMING and req.slack_seconds >= self.min_slack ] # Sort by highest slack (most ahead-of-time buffers) candidates.sort(key=lambda r: r.slack_seconds, reverse=True) freed_pages = 0 for cand in candidates: num_pages = len(cand.allocated_gpu_pages) if self.mem.swap_out_to_cpu(cand): freed_pages += num_pages if freed_pages >= needed_pages: return True return False

4. Running a Production Burst Benchmark

if __name__ == "__main__": # Simulate a GPU with 64 pages (1024 tokens capacity) mem_mgr = PagedKVCacheManager(total_gpu_pages=64, total_cpu_pages=256, page_size=16) scheduler = TokenFlowScheduler(mem_mgr, min_slack_for_preemption=2.0) # Initial long-generation streaming request req1 = StreamingRequest( request_id="stream_01_chat", arrival_time=0.0, prompt_tokens=128, max_output_tokens=500, client_read_rate=6.0 # 6 tokens/sec ) scheduler.add_request(req1) print("=== TokenFlow Serving Benchmark Simulation ===") current_time = 0.0 # Execute initial iterations to build up client buffer for step in range(5): batch = scheduler.schedule_iteration(current_time) for req in batch: if req.state == StreamState.PREFILLING: req.first_token_time = current_time req.state = StreamState.STREAMING # Engine generates 20 tokens per iteration req.generated_tokens += 20 current_time += 0.2 print(f"[Time {current_time:.1f}s] Stream 01 Status: Buffer = {req1.buffer_occupancy} tokens | Slack = {req1.slack_seconds:.2f}s") # SUDDEN BURST: Two short-prompt queries arrive simultaneously print("\n>>> BURST EVENT: 2 New Inquiries Arrive (VRAM Constrained) <<<") req_burst_A = StreamingRequest("burst_02_voice", arrival_time=current_time, prompt_tokens=256, max_output_tokens=64, client_read_rate=8.0) req_burst_B = StreamingRequest("burst_03_query", arrival_time=current_time, prompt_tokens=256, max_output_tokens=64, client_read_rate=7.0) scheduler.add_request(req_burst_A) scheduler.add_request(req_burst_B) # Schedule next tick batch = scheduler.schedule_iteration(current_time) print(f"[Time {current_time:.1f}s] Scheduled Execution Batch:") for b_req in batch: print(f" • {b_req.request_id:<16} | State: {b_req.state.value:<10} | GPU Pages: {len(b_req.allocated_gpu_pages)}") print(f"\nStream 01 State after Burst Scheduling: {req1.state.value}") print(f"Stream 01 Client Buffer Remaining: {req1.buffer_occupancy} tokens ({req1.slack_seconds:.2f}s playback remaining without stall!)")

Production Deployment: Integrating TokenFlow with vLLM and FastAPI

To deploy TokenFlow in enterprise Kubernetes clusters or automated data workflows through aiFetchly API Integration, TokenFlow acts as an intelligent inference proxy.

1. Reverse-Proxy Streaming Architecture

from fastapi import FastAPI, Request from fastapi.responses import StreamingResponse import asyncio app = FastAPI(title="TokenFlow Serving Gateway") @app.post("/v1/chat/completions") async def chat_streaming_endpoint(request: Request): payload = await request.json() client_id = payload.get("user", "default_user") async def event_generator(): # Connect to TokenFlow cluster proxy # Feedback channel reports client receive acknowledgements async for chunk in tokenflow_gateway.stream_response(payload): yield f"data: {chunk}\n\n" # Asynchronously send receive metrics back to scheduler tokenflow_gateway.report_consumption(client_id, len(chunk.split())) return StreamingResponse(event_generator(), media_type="text/event-stream")

Developers can configure automated web scraping pipelines to pipe extracted data into TokenFlow inference workers using aiFetchly Web Scraping Workflows.

Empirical Benchmark Results

Evaluated across open-weights models (Llama3-8B and Qwen2.5-32B) deployed across diverse hardware setups (NVIDIA RTX 4090, RTX A6000, and H200 SXM5), TokenFlow demonstrates remarkable gains under heavy Poisson request bursts:

P99 Time-To-First-Token (TTFT) Under 4x Request Burst: vLLM (Non-Preemptive Baseline): ████████████████████ 4,850 ms Sarathi-Serve (Chunked Prefill): ████████████ 2,920 ms TokenFlow (EuroSys 2026): ████ 720 ms (-80.2% Latency Reduction!) Effective Goodput Attainment (P99 TBT <= 200ms): vLLM Baseline: ██████████ 45.2 tok/s TokenFlow: ██████████████████ 82.5 tok/s (+82.5% Effective Goodput!)

Key Experimental Takeaways:

  1. 80.2% Reduction in P99 TTFT: By preempting requests that already hold large client buffers, newly arriving requests bypass queue delays and execute prefills immediately.
  2. Zero Text Streaming Stutter: Across 10,000 multi-turn test conversations, preempted clients experienced zero client-visible stalls, as all preemptions were resolved before local buffers reached zero.
  3. No Overall Throughput Degradation: Because asynchronous PCIe memory transfers overlap entirely with attention computation, TokenFlow preserves 100% of the underlying engine’s raw token generation throughput.

For complete deployment specifications and cluster hardware sizing calculators, consult the aiFetchly Documentation and the aiFetchly Sizing & Pricing Calculator.

Frequently Asked Questions (FAQ)

How does TokenFlow know the client’s reading or consumption speed?

TokenFlow dynamically measures consumption through client-side stream acknowledgements or applies conservative human reading baselines (e.g., $6\text{ tokens/s}$). For automated API consumers (such as code interpreters or agentic loops), clients can report their processing rate via an optional HTTP header.

What happens if a client reads faster than anticipated?

TokenFlow maintains a safety buffer margin ($\epsilon = 0.5\text{ s}$). If buffer slack drops below the safety threshold, the scheduler immediately raises the request’s priority to the top of the queue and swaps its KV cache back into GPU VRAM before the buffer empties.

Does TokenFlow require retraining the underlying foundation model?

No. TokenFlow is completely non-invasive and operates purely at the systems scheduling and memory management layer. It supports any standard Hugging Face, vLLM, or SGLang model checkpoint without fine-tuning.