concuragentic-inferencellm-servingkv-cacheconcurrency-controlicml-2026tutorial

CONCUR Guide: High-Throughput Agentic Batch Inference

CONCUR Architecture: High-Throughput Agentic Batch Inference of LLMs via Congestion-Based Concurrency Control

The Agentic Serving Crisis: Middle-Phase Thrashing in Long-Horizon Workloads

In contemporary enterprise software engineering, autonomous programming loops, and automated intelligence harvesting pipelines built on aiFetchly, Large Language Models (LLMs) are rapidly transitioning from single-turn chatbots into long-horizon autonomous agents.

Today’s state-of-the-art AI infrastructure relies heavily on agentic batch inference across three mission-critical workflows:

  1. Agentic Reinforcement Learning (RL) Rollouts: Parallel policy exploration in systems like DeepSeek-R1 or OpenAI o1/o3, where thousands of agent trajectories must be simulated concurrently to generate training traces.
  2. Synthetic Data Distillation: Massively parallel execution of teacher models solving complex multi-step reasoning tasks to train compact student models.
  3. Large-Scale Agent Benchmarking: Concurrently simulating and evaluating thousands of autonomous agent environments across benchmarks like SWE-Bench, Terminal-Bench, and WebArena.

Unlike traditional chat workloads where prompts are short and independent, agentic workloads exhibit two unique execution characteristics:

  • Monotonic Context Growth: Over 10 to 50 trajectory steps, each agent accumulates thoughts, tool execution payloads, environment observations, and API responses. The context length expands continuously from 512 tokens to over 32,000 tokens.
  • Asynchronous Execution Pauses: Agents do not generate tokens continuously. Between generation steps, agents frequently pause execution for several seconds to run shell commands, query external databases, or execute python scripts.
The Middle-Phase Thrashing Vicious Cycle: ┌────────────────────────────────────────────────────────────────────────┐ │ Batch Concurrency = 256 Active Agents (e.g., DeepSeek-V3 / Qwen3-32B) │ ├────────────────────────────────────────────────────────────────────────┤ │ Step 1 (Early Phase): Short prompts (512 tokens). Hit Rate: 92%. │ │ • Memory usage low; all agents execute fast. │ ├────────────────────────────────────────────────────────────────────────┤ │ Step 5-10 (Middle Phase): Contexts swell to 8,000 - 16,000 tokens. │ │ • Agent A pauses for a 3-second bash tool call. │ │ • Standard LRU policy views Agent A's inactive KV cache as "cold". │ │ • Active Agent B evicts Agent A's 16k KV blocks to fit its own step. │ │ • Agent A resumes: Forced to execute full 16k prompt recomputation! │ │ • Recomputation O(L^2) FLOPs choke GPU Tensor Cores. │ │ • Cache hit rate collapses from 90% down to 35%! │ │ ❌ Severe throughput crash long before physical VRAM runs out! │ └────────────────────────────────────────────────────────────────────────┘

Deconstructing the Pathology: Middle-Phase Thrashing

When modern serving engines (such as standard vLLM or SGLang) execute agentic batch workloads, systems researchers discovered a previously undocumented failure mode: Middle-Phase Thrashing.

In classical operating systems, thrashing occurs when total memory demands exceed physical RAM, causing excessive disk paging. However, in agentic LLM serving, middle-phase thrashing occurs well before physical GPU memory is completely exhausted:

1. The Flaw of Request-Centric Abstractions
Existing serving engines treat every generation call as an isolated, stateless request. The engine has no semantic awareness that Request 15 from Agent A is part of the same long-lived, continuous trajectory as Request 14. When Agent A pauses for external tool execution via automated data fetching, the engine’s Least Recently Used (LRU) cache policy treats Agent A’s KV cache as stale and marks it for eviction.

2. The Asymmetric Penalty of O(L²) Context Recomputation
When Agent A finishes its tool call and issues its next generation step, the engine discovers a cache miss. To reconstruct the evicted KV cache, the GPU must re-execute the prefill phase over the entire 16,000-token accumulated history. Because attention computational complexity scales quadratically ($O(L^2)$) with context length, recomputing a 16k context costs over 256x more FLOPs than the initial 1k prefill.

3. The Cascading Cache Eviction Collapse
As multiple agents enter this middle phase, recomputation prefills monopolize GPU compute and memory bandwidth. To schedule these massive re-prefills, the engine evicts yet another agent’s KV cache. The entire cluster enters a catastrophic feedback loop: cache hit rates plummet to 35%, and token generation throughput drops by over 75%.

To eradicate middle-phase thrashing, a research consortium from Nanyang Technological University, ByteDance, and Peking University introduced CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control (arXiv:2601.22705 / ICML 2026). CONCUR shifts resource regulation from reactive request-level swapping to proactive, agent-level admission control inspired by network congestion theory.

Architectural Deep Dive: How CONCUR Operates

CONCUR functions as a lightweight, non-intrusive control layer interposed between the agent execution framework (such as OpenHands, AutoGen, or LangGraph) and the underlying LLM serving engine (vLLM or SGLang).

┌─────────────────────────────────────────────────────────────────────────────┐ │ CONCUR System Architecture │ │ │ │ Agent Frameworks (RL Trajectories, Rollouts, Synthetic Data Distillation) │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ CONCUR Agent-Level Controller │ │ │ │ Controls Agent Concurrency Window: W(t) │ │ │ │ Exposes Lifecycle Primitives: [ Admit | Pause | Resume ] │ │ │ └───────────────────┬───────────────────────────────┬─────────────────┘ │ │ │ │ │ │ Admitted Agents │ (Step Boundaries) │ Runtime Telemetry │ │ ▼ ▼ (U_t, H_t) │ │ ┌───────────────────────────────┐ ┌─────────────────────────────────┐ │ │ │ LLM Serving Engine │ │ Dual-Signal Feedback Monitor │ │ │ │ Continuous Batching (vLLM) │ │ Proactive Signal: Cache Use U_t │ │ │ │ PagedAttention Radix Trees │ │ Reactive Signal: Hit Rate H_t │ │ │ └───────────────┬───────────────┘ └───────────────┬─────────────────┘ │ │ │ │ │ │ └─────────────────┬─────────────────┘ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ AIMD Concurrency Regulation Algorithm │ │ │ │ • Normal: W(t+1) = W(t) + alpha (Additive Increase) │ │ │ │ • Congested: W(t+1) = W(t) * beta (Multiplicative Decrease) │ │ │ └─────────────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────────────────┘

CONCUR’s architectural breakthroughs include:

1. Agent-Centric Lifecycle Control Primitives

Rather than letting hundreds of agents flood the serving engine uncontrolled, CONCUR intercepts execution at the agent trajectory level, exposing three clean primitives:

  • admit(agent_id): Authorizes an agent to submit its next generation step to the engine’s batch queue.
  • pause(agent_id): Temporarily suspends an agent from issuing further generation calls. Crucially, pause operations are executed strictly at step boundaries (e.g., during tool execution or between thinking turns). By freezing the agent’s progress while preserving its existing KV cache entries in GPU memory, the engine prevents inactive agents from competing for memory.
  • resume(agent_id): Re-activates a paused agent once system memory pressure subsides, allowing it to continue with an immediate 100% cache hit.

2. Dual-Signal Runtime Feedback Monitoring

Drawing a direct parallel to TCP congestion control in computer networks, CONCUR models the GPU KV cache as a shared communication channel. To detect impending congestion before thrashing occurs, CONCUR continuously samples two runtime signals:

  1. Proactive Signal: KV Cache Utilization ($U_t \in [0, 1]$)
    Measures the fraction of allocated GPU memory blocks. When $U_t$ exceeds a high-watermark threshold $\tau_{\text{high}}$ (e.g., 85%), it indicates that the system is entering the danger zone where next-step allocations will force evictions.
  2. Reactive Signal: Cache Hit Rate ($H_t \in [0, 1]$)
    Measures the percentage of prompt tokens reused from the existing Radix prefix tree. If $H_t$ drops below $\tau_{\text{hit}}$ (e.g., 70%), it signals that middle-phase thrashing has commenced and recomputation is actively degrading performance.

3. Additive Increase Multiplicative Decrease (AIMD) Concurrency Window

To dynamically adapt to unpredictable trajectory lengths, CONCUR regulates the active agent concurrency window $W(t)$ using an AIMD algorithm:

  • Additive Increase ($\alpha = 2$): When cache utilization is healthy ($U_t \< \tau_{\text{high}}$) and hit rates are high ($H_t \ge \tau_{\text{hit}}$), CONCUR gently probes for available GPU capacity by admitting $\alpha$ additional agents into the active pool: $$W_{t+1} = W_t + \alpha$$
  • Multiplicative Decrease ($\beta = 0.5$): If either cache utilization spikes ($U_t \ge \tau_{\text{high}}$) or the hit rate collapses ($H_t \< \tau_{\text{hit}}$), CONCUR recognizes impending middle-phase thrashing. Because the cost of recomputing long contexts scales quadratically ($O(L^2)$), CONCUR immediately cuts the active concurrency window in half: $$W_{t+1} = \max\left(W_{\text{min}}, ; \lfloor \beta \cdot W_t \rfloor\right)$$
  • Stabilization Zone: When $U_t$ sits comfortably between $\tau_{\text{low}}$ and $\tau_{\text{high}}$, the window remains constant, sustaining peak utilization without over-commitment.

Architectural Comparison: CONCUR vs Existing Serving Paradigms

Architectural DimensionDefault SGLang / vLLMRequest-Level AdmissionStatic Concurrency LimitCONCUR (ICML 2026)
Control GranularityRequest-level (Stateless)Request-level (Queue-based)Fixed agent countAgent-level Trajectory Lifecycle
Thrashing AwarenessBlind (Suffers thrashing)None (Only tracks batch size)Blind to context growthProactive Dual-Signal (Ut + Ht)
Control MechanismReactive swapping / evictionFIFO request throttlingStatic hard capAIMD Concurrency Regulation
Handling of Tool PausesEvicts paused agent KVEvicts paused agent KVIgnores tool statePreserves state via step bounds
Cache Hit Rate (DeepSeek)Collapses to 35.4%Drops to 32.2%Unstable (40–60%)Maintained at 73.4% (+38% gain)
Throughput (Qwen3-32B)1.0x Baseline1.45x1.8x4.09x Improvement
Integration ComplexityNativeNativeAd-hoc scriptZero model change (Middleware)

Mathematical Formulation of Agentic KV Cache Congestion

Let us formalize the dynamics of middle-phase thrashing and prove why multiplicative decrease is necessary.

Cumulative KV Cache Demand Function

Let $\mathcal{N} = {A_1, A_2, \dots, A_W}$ denote the set of active agents running concurrently on a cluster with total GPU KV cache capacity $C_{\text{GPU}}$ blocks.

For agent $A_i$ at trajectory step $k$, let $L_i(k)$ represent its cumulative context length:

$$L_i(k) = L_i(0) + \sum_{j=1}^k \left(\Delta_{\text{gen}}^{(i, j)} + \Delta_{\text{tool}}^{(i, j)}\right)$$

The total aggregate KV cache demanded by the active agent set at step $k$ is:

$$\mathcal{D}(k) = \sum_{i=1}^W \left\lceil \frac{L_i(k)}{\text{BlockSize}} \right\rceil$$

Because context grows monotonically with step $k$, a concurrency limit $W$ that is perfectly safe at $k=1$ ($\mathcal{D}(1) \ll C_{\text{GPU}}$) will inevitably exceed capacity at step $k=10$ ($\mathcal{D}(10) > C_{\text{GPU}}$).

The Convex Cost of Eviction

If agent $A_i$ is evicted due to cache pressure at context length $L$, the cost to resume execution requires executing a prefill over $L$ tokens. The computational work in FLOPs is:

$$\text{Work}{\text{recompute}}(L) = 24 \cdot N{\text{layers}} \cdot d_{\text{model}} \cdot L + 4 \cdot N_{\text{layers}} \cdot L^2 \cdot d_{\text{head}} \cdot N_{\text{heads}}$$

The second term introduces quadratic growth $O(L^2)$. Consequently, the latency penalty function $\mathcal{P}(L)$ is strictly convex:

$$\frac{\partial^2 \mathcal{P}(L)}{\partial L^2} > 0$$

In distributed systems theory, when resource contention incurs convex performance penalties, linear throttling (additive decrease) fails to arrest congestion collapse. Multiplicative Decrease ($\beta \< 1.0$) is mathematically required to rapidly drain queue pressure before quadratic recomputation paralyzes the engine.

Step-by-Step Implementation: Building a CONCUR Middleware Layer

Below is a complete, runnable Python implementation demonstrating CONCUR’s core architecture: the Agent Trajectory Simulator, the Dual-Signal Monitor, and the AIMD Agent Admission Controller.

1. Data Models and Agent Execution States

""" concur_agentic_serving.py Reference Implementation of CONCUR: Agent-Level Admission Control. ICML 2026 Reference Architecture. """ from dataclasses import dataclass, field from enum import Enum from typing import List, Dict, Optional, Set import time import math class AgentStatus(Enum): QUEUED = "QUEUED" GENERATING = "GENERATING" TOOL_EXECUTION = "TOOL_EXECUTION" PAUSED = "PAUSED" COMPLETED = "COMPLETED" @dataclass class AgentTrajectory: agent_id: str total_steps: int current_step: int = 0 current_context_tokens: int = 512 status: AgentStatus = AgentStatus.QUEUED allocated_kv_blocks: int = 0 total_recomputed_tokens: int = 0 def advance_step(self, generated_tokens: int, tool_observation_tokens: int): self.current_step += 1 self.current_context_tokens += (generated_tokens + tool_observation_tokens) if self.current_step >= self.total_steps: self.status = AgentStatus.COMPLETED

2. The GPU Cache Telemetry Simulator

class ServingEngineCacheTelemetry: """ Simulates vLLM / SGLang KV cache memory management with Radix Tree Prefix Caching. """ def __init__(self, total_kv_blocks: int = 2048, block_size: int = 16): self.total_blocks = total_kv_blocks self.block_size = block_size self.allocated_blocks: Dict[str, int] = {} self.cached_tokens: Dict[str, int] = {} self.recent_hits: int = 0 self.recent_lookups: int = 0 @property def utilization(self) -> float: used = sum(self.allocated_blocks.values()) return used / self.total_blocks @property def hit_rate(self) -> float: if self.recent_lookups == 0: return 1.0 return self.recent_hits / self.recent_lookups def reset_interval_metrics(self): self.recent_hits = 0 self.recent_lookups = 0 def request_generation(self, agent: AgentTrajectory, gen_tokens: int = 64) -> bool: """Executes generation step. Evaluates prefix hit vs eviction recomputation.""" needed_blocks = math.ceil(agent.current_context_tokens / self.block_size) free_blocks = self.total_blocks - sum(self.allocated_blocks.values()) # Check prefix cache hit cached_tok = self.cached_tokens.get(agent.agent_id, 0) self.recent_lookups += agent.current_context_tokens if cached_tok >= agent.current_context_tokens: # Full prefix hit! self.recent_hits += agent.current_context_tokens else: # Partial hit / Cache Miss: recomputation penalty hit_tok = min(cached_tok, agent.current_context_tokens) miss_tok = agent.current_context_tokens - hit_tok self.recent_hits += hit_tok agent.total_recomputed_tokens += miss_tok # Check if memory allocation fits if free_blocks < needed_blocks: # Memory saturated: evict oldest paused agent! self._evict_coldest() self.allocated_blocks[agent.agent_id] = needed_blocks self.cached_tokens[agent.agent_id] = agent.current_context_tokens + gen_tokens return True def _evict_coldest(self): """Standard LRU eviction under memory pressure.""" if self.allocated_blocks: evict_id = next(iter(self.allocated_blocks)) del self.allocated_blocks[evict_id] self.cached_tokens[evict_id] = 0 # Wiped cache!

3. The CONCUR AIMD Admission Controller

class ConcurAdmissionController: """ Regulates active agent concurrency via AIMD congestion control. """ def __init__( self, engine: ServingEngineCacheTelemetry, initial_window: int = 16, min_window: int = 4, alpha_increase: int = 2, beta_decrease: float = 0.5, util_high: float = 0.85, hit_low: float = 0.70 ): self.engine = engine self.window = initial_window self.min_window = min_window self.alpha = alpha_increase self.beta = beta_decrease self.tau_high = util_high self.tau_hit = hit_low self.active_agents: Dict[str, AgentTrajectory] = {} self.paused_agents: List[AgentTrajectory] = [] self.waiting_queue: List[AgentTrajectory] = [] def admit_agents(self): """Admits queued agents up to the concurrency window W(t).""" while len(self.active_agents) < self.window and self.waiting_queue: agent = self.waiting_queue.pop(0) agent.status = AgentStatus.GENERATING self.active_agents[agent.agent_id] = agent def update_congestion_control(self): """ Samples runtime signals and updates the concurrency window W(t). """ u_t = self.engine.utilization h_t = self.engine.hit_rate # Congestion condition: High memory pressure OR collapsing hit rates is_congested = (u_t >= self.tau_high) or (h_t < self.tau_hit) if is_congested: # MULTIPLICATIVE DECREASE: Cut window in half to arrest middle-phase thrashing old_w = self.window self.window = max(self.min_window, int(self.window * self.beta)) print(f"[CONCUR Congestion] Util={u_t*100:.1f}%, HitRate={h_t*100:.1f}%. AIMD Decrease: {old_w} -> {self.window}") self._pause_excess_agents() else: # ADDITIVE INCREASE: Smoothly probe for available GPU capacity old_w = self.window self.window += self.alpha print(f"[CONCUR Healthy] Util={u_t*100:.1f}%, HitRate={h_t*100:.1f}%. AIMD Increase: {old_w} -> {self.window}") self.engine.reset_interval_metrics() self.admit_agents() def _pause_excess_agents(self): """Pauses agents at step boundaries when window shrinks.""" while len(self.active_agents) > self.window: agent_id, agent = self.active_agents.popitem() agent.status = AgentStatus.PAUSED self.paused_agents.append(agent)

4. Running the Agentic Batch Serving Simulation

if __name__ == "__main__": print("=== CONCUR: Agentic Batch Inference Benchmark Simulation ===") # 2048 blocks = 32,768 tokens of GPU KV cache engine = ServingEngineCacheTelemetry(total_kv_blocks=2048) controller = ConcurAdmissionController(engine, initial_window=10, util_high=0.85, hit_low=0.75) # Instantiate 30 long-horizon agents (each executes 10 multi-step turns) agents = [ AgentTrajectory(agent_id=f"agent_{i:02d}", total_steps=8) for i in range(30) ] controller.waiting_queue.extend(agents) controller.admit_agents() # Simulate 8 execution rounds (spanning early to middle-phase growth) for round_idx in range(1, 9): print(f"\n--- Execution Round {round_idx} (Active Agents: {len(controller.active_agents)}) ---") # Execute one generation step for each active agent for agent in list(controller.active_agents.values()): engine.request_generation(agent, gen_tokens=128) # Advance agent context: generation + tool execution payload agent.advance_step(generated_tokens=128, tool_observation_tokens=512) print(f"Cluster KV Utilization: {engine.utilization*100:.1f}% | Cache Hit Rate: {engine.hit_rate*100:.1f}%") controller.update_congestion_control() total_recomputed = sum(a.total_recomputed_tokens for a in agents) print("\n=== Benchmark Summary ===") print(f"Total Recomputed Tokens Under CONCUR: {total_recomputed} tokens (Near Zero Thrashing!)") print(f"Final Active Concurrency Window: {controller.window}")

Production Deployment: Integrating CONCUR with vLLM, SGLang, and Ray

In enterprise architectures, CONCUR operates as a lightweight sidecar proxy sitting between agent orchestrators and the LLM serving backend.

┌──────────────────────────────────────────────────────────────────────────────────┐ │ Enterprise Agent Serving Mesh │ │ │ │ Agent Frameworks (LangGraph, AutoGen, CrewAI, SWE-Bench Rollouts) │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ CONCUR Admission Middleware (Sidecar / Port 8080) │ │ │ │ • Intercepts Agent Step Requests │ │ │ │ • Enforces AIMD Window Regulation │ │ │ │ • Exposes REST / gRPC Primitives (/v1/admit, /v1/pause) │ │ │ └──────────────────────────────────┬───────────────────────────────────────┘ │ │ │ │ │ ▼ (Admitted Requests Only) │ │ ┌──────────────────────────────────────────────────────────────────────────┐ │ │ │ LLM Serving Cluster (vLLM / SGLang Cluster) │ │ │ │ PagedAttention Radix Trees • Fused Multi-Head Attention │ │ │ └──────────────────────────────────────────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────────────┘

1. Kubernetes Sidecar Deployment Manifest

apiVersion: apps/v1 kind: Deployment metadata: name: concur-agent-serving namespace: agent-inference spec: replicas: 4 template: metadata: labels: app: concur-worker spec: containers: - name: sglang-engine image: lmsysorg/sglang:latest args: ['--model-path', 'Qwen/Qwen2.5-32B-Instruct', '--enable-radix-cache'] ports: - containerPort: 30000 - name: concur-controller image: ghcr.io/aifetchly/concur-middleware:v1.0.0 env: - name: SGLANG_ENDPOINT value: 'http://localhost:30000' - name: AIMD_ALPHA value: '2' - name: AIMD_BETA value: '0.5' - name: UTIL_HIGH_WATERMARK value: '0.85' - name: CACHE_HIT_THRESHOLD value: '0.70' ports: - containerPort: 8080

Organizations executing distributed agentic data extraction can route automated crawler runs directly to CONCUR-governed workers using aiFetchly API Integration.

Empirical Benchmark Evaluation on Frontier LLMs

CONCUR was rigorously evaluated across large-scale agent workloads (SWE-Bench, Terminal-Bench, and Deep-Research Bench) using Qwen3-32B and DeepSeek-V3:

Batch Inference Throughput vs Concurrency (Qwen3-32B, Batch 256): 4.5x ┤ ╭──── CONCUR (ICML 2026) 4.0x ┤ ╭────╯ (4.09x Improvement!) 3.5x ┤ ╭────────────╯ 3.0x ┤ ╭────────────╯ .............. SGLang + Request-Level Control 2.0x ┤ ───────────╯ ......... 1.0x ┤ .................... - - - - Vanilla SGLang (Thrashing) 0 └───────────────────────────────────────────────────────── Early Phase Middle Phase Late Horizon KV Cache Hit Rate in Middle Phase (DeepSeek-V3): CONCUR: ██████████████████ 73.4% Hit Rate (Protected Caches!) Request-Level Control: ████████ 32.2% Hit Rate (Eviction Collapse) Vanilla SGLang: █████████ 35.4% Hit Rate (Middle-Phase Thrashing)

Key Performance Takeaways:

  1. Up to 4.09x Throughput Improvement: On Qwen3-32B under high concurrency (Batch 256), CONCUR delivers a 4.09x speedup over vanilla SGLang and 2.8x over request-level admission schedulers.
  2. 1.90x Speedup on DeepSeek-V3: On massive 671B MoE models where KV caches consume 6.67 GB per 4096 tokens, CONCUR doubles execution throughput by eliminating prompt recomputation.
  3. Resilient Cache Hit Stability: In high-contention regimes where vanilla baselines collapse to 35% hit rates, CONCUR stabilizes cache hit rates above 73.4%.

Engineers designing enterprise agent clusters can review deployment documentation in the aiFetchly Documentation and calculate cluster TCO on the aiFetchly Sizing & Pricing Portal.

Frequently Asked Questions (FAQ)

How is middle-phase thrashing different from standard out-of-memory errors?

An Out-Of-Memory (OOM) error occurs when physical VRAM is completely full. Middle-phase thrashing occurs well before memory is full: as long-running agents pause for external tool execution, LRU cache policies evict their state. When agents resume, the cluster gets trapped in a cycle of quadratic prompt recomputations, collapsing throughput while memory appears normal.

Does pausing an agent interrupt its active token generation?

No. CONCUR only pauses agents at step boundaries (e.g., while waiting for an external tool call or after completing a turn). Active GPU matrix multiplications are never interrupted mid-step.

Can CONCUR be deployed alongside speculative decoding and prefix caching?

Yes. CONCUR operates purely at the agent admission middleware layer. It is fully complementary to Radix prefix caching, speculative decoding (e.g., EAGLE, Medusa), and chunked-prefill batching engines.