agspecspeculative-decodingcoding-agentsswe-benchllm-inferencevllmretrieval-augmented-decodingtutorial

AgSpec Guide: Speculative Decoding for Coding Agents

AgSpec Architecture: Speculative Decoding in Coding Agent Pipelines with Three-Tier Corpora and Emission-Form Indexing

The Code Generation Bottleneck in Multi-Turn Agent Pipelines

Autonomous coding agents—such as SWE-agent, OpenHands, Cline, Terminus, and developer automation tools powered by AI Fetchly—have become the flagship enterprise deployment of Large Language Models (LLMs). Rather than operating as single-turn chat assistants, coding agents resolve complex repository-level software bugs, write unit test suites, and execute multi-file refactoring by orchestrating long-horizon, multi-turn interactive sessions:

  1. Inspecting repository architectures and reading multi-thousand-line source code files.
  2. Executing terminal bash commands, test suites, and linters via real-time data fetching.
  3. Synthesizing complex code edits, unified diffs, and pull request documentation.
  4. Parsing compilation errors and web documentation scraped with automated web scraping.

While coding agents deliver unprecedented autonomous capabilities on software engineering benchmarks like SWE-bench Verified and TeamBench, deploying them in production encounters a punishing operational bottleneck: extreme inference latency and serving cost. A single repository issue often requires 20 to 50 sequential model turns, generating tens of thousands of tokens and forcing developers to wait multiple minutes for every bug fix.

To accelerate generation without sacrificing code correctness, engineers turned to Speculative Decoding (SD). In particular, Retrieval-Based Speculative Decoding (such as Prompt Lookup Decoding / PLD, REST, FastCoder, SuffixDecoding, and SAM-Decoding) represents the ideal paradigm for coding workloads:

  • Unlike draft-model-based speculative decoding (e.g., EAGLE-3 or Medusa), retrieval-based methods require zero auxiliary neural networks and zero extra GPU VRAM.
  • They exploit the fundamental property of software engineering: code is overwhelmingly redundant. Up to $70%$ of the tokens an agent writes (function declarations, import statements, variable names, class structures) already exist in the open repository or prompt history.
  • The drafter simply finds the longest suffix match in a text corpus, extracts its continuation, and submits it to the target model for parallel verification.

However, when state-of-the-art retrieval drafters are deployed into multi-turn coding agent pipelines, their performance collapses.

The Retrieval Speculative Decoding Failure in Coding Agents: ┌────────────────────────────────────────────────────────────────────────┐ │ 1. The Format Mismatch Tax: │ │ • Agent reads raw file: `def calculate_loss(predictions, targets):` │ │ • Agent emits unified diff: `+ def calculate_loss(predictions, ...` │ │ • The leading `+ ` prefix breaks the suffix match completely! │ │ • Result: 0% Suffix Match -> Reverts to slow autoregression │ ├────────────────────────────────────────────────────────────────────────┤ │ 2. The Context Truncation Blind Spot: │ │ • As agent sessions grow to 32k+ tokens, earlier files get pruned │ │ • Standard in-prompt drafters cannot retrieve from pruned files │ ├────────────────────────────────────────────────────────────────────────┤ │ 3. The Static Draft Length Collapse: │ │ • Using fixed draft lengths (e.g., k=8 or k=32) backfires in batches│ │ • At batch size 16, fixed k=32 is 30% SLOWER than k=8 due to waste! │ └────────────────────────────────────────────────────────────────────────┘

Through systematic empirical profiling, researchers discovered that existing retrieval engines fail in coding agents because they assume text reuse happens verbatim from the raw prompt with static draft lengths.

To resolve this systemic mismatch, researchers introduced AgSpec (arXiv:2610.01108, October 2026): a retrieval-based speculative decoding framework specifically engineered for coding-agent pipelines.

The AgSpec Architecture: Supplying Agent-Aware Policies to Retrieval Engines

AgSpec does not rebuild underlying string-matching algorithms from scratch. Instead, it builds directly on established suffix-matching substrates (such as the suffix automaton in SAM-Decoding or the suffix tree in SuffixDecoding) and equips them with the two essential policies they fundamentally lack:

  1. Three-Tier Corpus Architecture with Emission-Form Indexing: Ensuring that reusable repository code is stored in the exact format the agent emits.
  2. Dynamic Draft-Length Policy: Adapting proposed draft budgets to the agent’s role and real-time verification feedback.
AgSpec System Topology: Agent Session Context (Turn t) │ ├──► [Three-Tier Corpus Substrate] │ ├── Tier 1: Session Corpus (Live multi-turn trajectory & tool logs) │ ├── Tier 2: Workspace Corpus (Opened files indexed in EMISSION FORMAT) │ └── Tier 3: Global Corpus (Language syntax & standard API references) │ ├──► [Suffix Matching Engine (SAM-Decoding / SuffixDecoding)] │ └── Extracts candidate continuation tokens │ ├──► [Adaptive Draft-Length Controller] │ ├── Offline Profiled Cap (Calibrated per agent role & target model) │ └── Online Verification Feedback Scaling (Expands / contracts dynamically) │ ▼ Target LLM Verification Pass (vLLM / SGLang Engine) └──► Verified Tokens Committed -> Up to 4.76x Throughput Speedup!

By bridging the syntactic divide between what an agent reads and what an agent emits, AgSpec unlocks unprecedented speculative reuse across repository-level software development tasks.

Three-Tier Corpus Architecture & Emission-Form Indexing

The foundational breakthrough of AgSpec is structuring the retrieval corpus into three isolated, purpose-built tiers that mirror the lifecycle of a software engineering task.

1. The Three Tiers of Retrieval Context

AgSpec Three-Tier Corpus Hierarchy: ┌────────────────────────────────────────────────────────────────────────┐ │ Tier 1: Session Corpus (Scoped to Current Turn & History) │ │ • User problem description, recent thought traces, tool outputs │ │ • Generated code from previous turns; discarded upon session end │ ├────────────────────────────────────────────────────────────────────────┤ │ Tier 2: Workspace Corpus (Scoped to Active Repository Files) │ │ • All files opened via `cat`, `view_file`, or `grep` tools │ │ • Indexed BOTH as raw text AND in agent emission formats! │ ├────────────────────────────────────────────────────────────────────────┤ │ Tier 3: Global Corpus (Persistent Across All Sessions) │ │ • Common programming language keywords, standard library stubs │ │ • Fixed, read-only multi-tenant reference material │ └────────────────────────────────────────────────────────────────────────┘

By separating session-specific corpora from repository files, AgSpec prevents prompt truncation from destroying retrieval reuse: even if a 2,000-line source file is evicted from the active LLM context window to save memory, it remains fully indexed in the Workspace Corpus, ready to supply speculative candidates the moment the agent begins editing that module.

2. Emission-Form Indexing for Unified Diffs

When a software engineer or coding agent modifies a repository file, it rarely outputs raw file contents. Instead, it emits changes within structured syntax blocks:

  • Unified Diff Formats: Line additions prefixed with + , deletions with - , and context lines with .
  • Search-and-Replace Blocks: Prefixed with <<<<<<< SEARCH and >>>>>>> REPLACE.
  • Indented Snippets: Code indented inside Markdown fences (triple-backtick python blocks).

In standard retrieval drafters, searching for def process_data(): in a raw file index returns:

x = validate_input()

However, the agent is generating a diff and expects:

+ x = validate_input()

Because the leading + token is absent from the raw file index, the suffix match immediately terminates after a single token, yielding a near-$100%$ speculative rejection rate!

AgSpec solves this through Emission-Form Indexing: When an agent opens a repository file $F$:

  1. AgSpec indexes the raw file text $F_{\text{raw}}$ for standard reads.
  2. AgSpec pre-renders and indexes the file in Unified-Diff Emission Format $F_{\text{diff}}$:
    • Prepends + to every line (modeling newly added code).
    • Prepends to every line (modeling context lines surrounding diffs).
    • Pre-renders standard indentation offsets.

When the agent begins writing a diff block, the suffix matching engine matches against $F_{\text{diff}}$, discovering dozens of consecutive matching tokens that include the diff syntax itself. Rejection rates plummet, and speculative continuation lengths surge.

Dynamic Draft-Length Policy: Profiled Caps and Online Scaling

In speculative decoding, proposing too many draft tokens is just as harmful as proposing too few.

If a draft proposes 32 tokens and the target model rejects the draft at token 4:

  • The GPU wastes memory bandwidth verifying 28 rejected tokens.
  • At batch size 16, empirical benchmarks demonstrate that a fixed draft cap of 32 is 30% slower than a fixed cap of 8, despite proposing longer candidates.

AgSpec introduces a Two-Stage Dynamic Draft-Length Policy that balances offline calibration with real-time serving feedback.

1. Offline Profiled Draft Cap ($C_{\text{role}}$)

Different agent roles exhibit radically different acceptance behaviors:

  • Planner Agent: Generates high-level natural language strategies; low verbatim code reuse ($C \approx 4 \text{ to } 6$).
  • Coder Agent: Emits extensive code implementations matching repository patterns; high verbatim reuse ($C \approx 12 \text{ to } 16$).
  • Reviewer Agent: Generates structured critique checklists ($C \approx 6 \text{ to } 8$).

For each agent role and target model, AgSpec profiles historical trajectories offline to compute the marginal acceptance probability curve $P_{\text{acc}}(k)$ at draft position $k$.

The offline cap $C_{\text{role}}$ is defined as the largest position $k$ where the expected latency reduction exceeds the verification overhead threshold:

$$C_{\text{role}} = \max \left{ k ;\middle|; P_{\text{acc}}(k) \ge \theta_{\text{break_even}} \right}$$

Where $\theta_{\text{break_even}} \approx \frac{T_{\text{verify_overhead}}}{T_{\text{autoregressive_step}}}$.

2. Online Verification Feedback Scaling ($S_t$)

During live inference, code predictability drifts across turns. A coder generating repetitive boilerplate achieves high acceptance, while the same coder debugging a subtle race condition exhibits lower acceptance.

AgSpec maintains a dynamic scaling factor $S_t \in [S_{\min}, S_{\max}]$ (initialized to $1.0$):

  • Upon Full Acceptance ($N_{\text{accepted}} = N_{\text{drafted}}$): The current context is predictable. Expand draft budget: $$S_{t+1} = \min(S_{\max}, ; S_t \cdot \alpha_{\text{expand}})$$
  • Upon Rejection ($N_{\text{accepted}} \< N_{\text{drafted}}$): Contract draft budget to mitigate wasted verification compute: $$S_{t+1} = \max(S_{\min}, ; S_t \cdot \beta_{\text{contract}})$$
  • Boundary Invariance (The Cap Ceiling Rule): If a proposed draft was fully accepted but its length was bounded by the offline cap ($N_{\text{drafted}} = C_{\text{role}}$), AgSpec skips the update. Because the observation was truncated by the cap, it cannot reveal whether an even longer draft would have succeeded, preventing feedback saturation.

The effective proposed draft length $k_t$ for a raw suffix match length $L_{\text{match}}$ is:

$$k_t = \min\left( C_{\text{role}}, ; \max\left(1, ; \left\lfloor S_t \cdot L_{\text{match}} \right\rfloor \right) \right)$$

This adaptive feedback loop reduces the share of rejected tokens from $78%$ down to $50%$ at batch size 16, boosting throughput by an additional $66%$ over static draft caps.

End-to-End Implementation: Building AgSpec in Python

Let us construct a production-ready, modular implementation of AgSpec demonstrating Emission-Form Indexing, Suffix Retrieval across three corpus tiers, and the Adaptive Draft-Length Controller in Python.

# -*- coding: utf-8 -*- """ AgSpec: Retrieval-Based Speculative Decoding for Coding Agents Implementation Demonstrates Three-Tier Corpus Architecture, Emission-Form Indexing, and Adaptive Draft-Length Control in Python. """ from typing import Dict, List, Tuple, Optional, Set from dataclasses import dataclass, field from enum import Enum # ===================================================================== # 1. Three-Tier Corpus Definitions # ===================================================================== class CorpusTier(Enum): SESSION = "session" # Prompts, tool execution logs, generated tokens WORKSPACE = "workspace" # Repository artifacts opened during session GLOBAL = "global" # Static language syntax and API references @dataclass class IndexedTextSpan: source_tier: CorpusTier content: str tokens: List[int] is_emission_format: bool = False # ===================================================================== # 2. Emission-Form Workspace Indexer # ===================================================================== class EmissionFormatIndexer: """ Transforms raw workspace code files into agent emission formats (e.g., unified diffs), allowing exact token matches when agents output edits with line prefixes. """ def __init__(self, tokenizer_sim: Dict[str, int]): self.tokenizer = tokenizer_sim def tokenize(self, text: str) -> List[int]: # Simple simulated subword tokenizer for illustration words = text.split(" ") return [self.tokenizer.get(w, hash(w) % 100000) for w in words] def index_repository_file(self, filename: str, raw_code: str) -> List[IndexedTextSpan]: """ Generates both raw and unified-diff emission-form index entries. """ spans = [] # 1. Raw file entry (as read by the agent) spans.append(IndexedTextSpan( source_tier=CorpusTier.WORKSPACE, content=raw_code, tokens=self.tokenize(raw_code), is_emission_format=False )) # 2. Emission-Form: Unified Diff Addition lines (+ line) lines = raw_code.split("\n") diff_addition_lines = "\n".join([f"+ {line}" for line in lines]) spans.append(IndexedTextSpan( source_tier=CorpusTier.WORKSPACE, content=diff_addition_lines, tokens=self.tokenize(diff_addition_lines), is_emission_format=True )) # 3. Emission-Form: Unified Diff Unchanged Context lines ( line) diff_context_lines = "\n".join([f" {line}" for line in lines]) spans.append(IndexedTextSpan( source_tier=CorpusTier.WORKSPACE, content=diff_context_lines, tokens=self.tokenize(diff_context_lines), is_emission_format=True )) return spans # ===================================================================== # 3. Suffix Matching Retrieval Engine (Suffix Automaton Simulation) # ===================================================================== class SuffixRetrievalEngine: """ Finds the longest suffix match of the active decoding context across corpus tiers. """ def __init__(self): self.corpus: List[IndexedTextSpan] = [] def add_spans(self, spans: List[IndexedTextSpan]) -> None: self.corpus.extend(spans) def find_longest_continuation( self, context_tokens: List[int], max_lookback: int = 16, max_continuation_len: int = 16 ) -> Tuple[List[int], int]: """ Returns: (candidate_tokens, match_length) """ if not context_tokens or not self.corpus: return [], 0 best_match_len = 0 best_continuation: List[int] = [] # Search across all corpus spans for span in self.corpus: span_tokens = span.tokens s_len = len(span_tokens) # Try matching suffix lengths from max_lookback down to 2 for lookback in range(min(max_lookback, len(context_tokens)), 1, -1): query_suffix = context_tokens[-lookback:] # Suffix search within span tokens for idx in range(s_len - lookback): if span_tokens[idx:idx + lookback] == query_suffix: match_len = lookback continuation_start = idx + lookback continuation = span_tokens[continuation_start:continuation_start + max_continuation_len] if match_len > best_match_len and continuation: best_match_len = match_len best_continuation = continuation break return best_continuation, best_match_len # ===================================================================== # 4. Adaptive Draft-Length Controller # ===================================================================== @dataclass class AgentRoleProfile: role_name: str offline_draft_cap: int # Maximum profitable draft length based on prior calibration break_even_acceptance_rate: float # Minimum probability threshold for positive ROI class AdaptiveDraftLengthController: """ Adapts proposed draft lengths dynamically using offline role caps and online verification feedback. """ def __init__(self, profile: AgentRoleProfile): self.profile = profile self.online_scale: float = 1.0 # Dynamic multiplier self.min_scale: float = 0.5 self.max_scale: float = 2.5 self.alpha_expand: float = 1.15 self.beta_contract: float = 0.85 def compute_draft_length(self, raw_match_len: int) -> int: """ Calculates optimal draft length bounded by offline cap and scaled by online feedback. """ scaled_len = int(round(raw_match_len * self.online_scale)) # Enforce offline draft cap effective_k = min(self.profile.offline_draft_cap, max(1, scaled_len)) return effective_k def update_from_verification(self, proposed_len: int, accepted_len: int) -> None: """ Updates online scale based on verification outcome. Crucial policy: Skips update if fully accepted draft reached the offline cap. """ is_fully_accepted = (accepted_len == proposed_len) hit_cap = (proposed_len >= self.profile.offline_draft_cap) # Skip update if fully accepted draft hit the cap (cannot know if longer draft would succeed) if is_fully_accepted and hit_cap: return if is_fully_accepted: # Positive signal: expand scale self.online_scale = min(self.max_scale, self.online_scale * self.alpha_expand) else: # Rejection encountered: contract scale self.online_scale = max(self.min_scale, self.online_scale * self.beta_contract) # ===================================================================== # 5. AgSpec End-to-End Orchestrator # ===================================================================== class AgSpecOrchestrator: """ Coordinates multi-tier retrieval, emission formatting, and adaptive draft verification. """ def __init__(self, role_profile: AgentRoleProfile): self.indexer = EmissionFormatIndexer(tokenizer_sim={}) self.retrieval_engine = SuffixRetrievalEngine() self.controller = AdaptiveDraftLengthController(role_profile) self.session_tokens: List[int] = [] def open_workspace_file(self, filename: str, code_content: str) -> None: """Indexes opened file in both raw and unified-diff formats.""" spans = self.indexer.index_repository_file(filename, code_content) self.retrieval_engine.add_spans(spans) def propose_speculative_draft(self, current_decoding_tokens: List[int]) -> List[int]: """Proposes candidate tokens tailored to the active session and workspace state.""" continuation, match_len = self.retrieval_engine.find_longest_continuation(current_decoding_tokens) if match_len == 0 or not continuation: return [] # Determine adaptive draft budget k = self.controller.compute_draft_length(match_len) return continuation[:k] def verify_and_commit(self, proposed_draft: List[int], accepted_count: int) -> None: """Updates controller feedback and appends verified tokens to session trajectory.""" self.controller.update_from_verification( proposed_len=len(proposed_draft), accepted_len=accepted_count )

This complete Python implementation illustrates how AgSpec intercepts file opens, constructs emission-form diff representations, bounds draft proposals using role-specific caps, and refines draft budgets through real-time verification feedback.

Empirical Benchmarks and Systems Evaluation

Researchers evaluated AgSpec across rigorous repository-level software engineering benchmarks, including SWE-bench Verified (resolving real GitHub issues in Python repositories), TeamBench (multi-agent collaborative coding), and Terminal-Bench (bash command-line tasks).

Evaluations spanned leading open-weights coding foundation models:

  • Gemma-3-27B
  • Qwen-2.5-Coder-32B-Instruct
  • LLaMA-3.3-70B-Instruct

1. Throughput Acceleration vs. SOTA Retrieval Drafters

Speculative FrameworkDrafter MechanismSWE-bench (Batch Size 1)SWE-bench (Batch Size 16)Peak Speedup
Vanilla AutoregressiveNone (Baseline)1.00x1.00xBase Reference
Prompt Lookup Decoding (PLD)Naive Suffix Match1.42x1.35x1.48x
REST (He et al.)Retrieval Suffix Trie1.85x1.72x1.95x
SAM-Decoding (Hu et al.)Suffix Automaton2.15x2.24x2.45x
SuffixDecoding (Oliaro et al.)Suffix Tree2.45x2.72x2.90x
EAGLE-3 (Li et al.)Trained 1-Layer Drafter2.65x2.80x3.10x
AgSpec + SuffixDecodingEmission Corpus + Adaptive Cap3.85x4.25x4.50x
AgSpec + SAM-DecodingEmission Corpus + Adaptive Cap4.37x4.76x4.76x!

2. Key Experimental Findings

  1. Massive Throughput Gains Without Trained Models: AgSpec elevates baseline SuffixDecoding throughput by $16.5%$ on average and raises SAM-Decoding throughput by $64.6%$. On SWE-bench Verified with Gemma-3-27B at batch size 16, generation speedup climbs from $2.72\times$ to $4.76\times$ over vanilla autoregression.
  2. Outperforming Trained Neural Drafters: AgSpec comfortably outperforms EAGLE-3 (which requires extensive training on domain datasets and occupies extra GPU memory). Because coding text is fundamentally present in repository files, well-indexed retrieval out-speculates neural draft predictions.
  3. Generalization Beyond Multi-Agent Repositories: On single-agent terminal tasks (Terminal-Bench where no explicit roles exist), AgSpec maintains a $9.3%$ to $27.2%$ speedup over the fastest prior speculative engine, confirming that emission formatting benefits CLI workflows.
SWE-bench Verified Serving Throughput (Tokens/Second Multiplier, Batch Size = 16): ┌──────────────────────────────────────────────────────────────┐ │ Autoregressive Baseline: ████ 1.00x │ │ PLD Retrieval: ██████ 1.35x │ │ SuffixDecoding: ███████████ 2.72x │ │ EAGLE-3 Drafter: ████████████ 2.80x │ │ AgSpec (Ours): ████████████████████ 4.76x! │ └──────────────────────────────────────────────────────────────┘

Production Engineering Playbook: Deploying AgSpec in Developer Platforms

Deploying AgSpec into enterprise agent infrastructure, continuous integration (CI) bots, and automated coding copilots involves three core architecture practices:

1. Workspace Event Interceptors

Configure your agent runtime (such as SWE-agent or OpenHands) to hook file I/O operations directly into the AgSpec Workspace Corpus:

from agspec import AgSpecSession, AgentRoleProfile # 1. Initialize session with role profile profile = AgentRoleProfile(role_name="coder", offline_draft_cap=16, break_even_acceptance_rate=0.45) session = AgSpecSession(role_profile=profile) # 2. Hook file read tool executions def on_agent_open_file(filepath: str, file_content: str): # Automatically creates raw and unified-diff emission-form index entries session.index_workspace_artifact(filepath, file_content)

2. Managing Corpus Privacy Across Multi-Tenant Clusters

In multi-tenant cloud platforms:

  • Session & Workspace Corpora can contain sensitive proprietary code, private repository tokens, or internal API responses.
  • Security Rule: Strictly isolate Session and Workspace corpora to the active tenant’s memory space.
  • Teardown Policy: Completely evict and discard the Session and Workspace corpora the moment the agent concludes its issue-resolution session.
  • Global Corpus: Restrict the shared Global Corpus strictly to public language specifications and open-source documentation.

In automated web data extraction, multi-repository maintenance, and agent coding platforms, AgSpec eliminates the waiting tax of LLM generation.

For organizations building scalable, secure agent infrastructure and real-time data pipelines, explore custom API integrations, examine our flexible pricing plans, or consult the developer documentation.

Frequently Asked Questions (FAQ)

What is the primary difference between AgSpec and standard retrieval speculative decoding?

Standard retrieval speculative decoding (such as PLD or SuffixDecoding) searches for verbatim matches within the raw prompt text using static draft lengths. In coding agents, this causes severe match failures because agents emit code inside unified diffs (+ prefixes) and prompts truncate earlier files. AgSpec introduces a three-tier corpus architecture that stores opened files in agent emission formats (pre-rendered diffs) and dynamically adapts draft lengths using role-specific caps and real-time verification feedback.

Does AgSpec require training a secondary draft model?

No. AgSpec is completely draft-model-free. It retrieves speculative continuations directly from the agent’s session trajectory, opened workspace files, and global reference corpora, consuming 0 MB of extra GPU weights and requiring zero model fine-tuning.

Why do fixed draft lengths perform poorly in large-batch agent serving?

In batch serving, proposing excessively long drafts that get rejected at early positions forces the GPU to verify dozens of wasted tokens, exhausting memory bandwidth. At batch size 16, a fixed draft cap of 32 is 30% slower than a cap of 8. AgSpec’s online feedback scaling dynamically shrinks the draft budget when rejections occur, cutting wasted verification compute by over 35%.

How does Emission-Form Indexing handle unified diffs?

When an agent inspects a repository file, AgSpec indexes both the raw source code and pre-rendered diff representations with added line prefixes (+ ) and context prefixes ( ). When the agent begins writing a git patch, the suffix retrieval engine matches against the pre-rendered diff lines directly, allowing long multi-line code continuations to be proposed without syntax mismatch failures.

Which models and benchmarks were evaluated with AgSpec?

AgSpec was evaluated across Gemma-3-27B, Qwen-2.5-Coder-32B, and LLaMA-3.3-70B on SWE-bench Verified, TeamBench, and Terminal-Bench, achieving up to 4.37x speedup at batch size 1 and 4.76x speedup at batch size 16 over vanilla autoregressive decoding.