EAGLE-2: Feature-Level Speculative Decoding Guide
The Problem: The Inefficiency of Token-Level Speculative Decoding
Speculative decoding is one of the most effective strategies for accelerating Large Language Model (LLM) inference. By using a lightweight drafting mechanism to propose a sequence of candidate tokens and verifying them in a single batched forward pass through the target LLM, speculative decoding breaks the sequential memory-bandwidth latency wall.
However, early speculative decoding paradigms suffer from severe structural bottlenecks that cap real-world speedups:
1. The Distribution Shift of Separate Draft Models
In classical speculative sampling (Leviathan et al., Chen et al.), the draft engine is an independent, smaller foundation model (e.g., pairing a LLaMA-68M or LLaMA-1B draft model with a LLaMA-70B target model).
This approach introduces severe distribution shift:
- Even when trained on similar data, the small draft model’s probability distribution diverges substantially from the large target model.
- On complex mathematical reasoning, multi-turn dialogue, and code generation, the draft model’s acceptance rate drops to 40% to 55%.
- When draft tokens are rejected, the target model discards the proposals, resulting in wasted drafting compute and negligible net wall-clock speedups.
2. The Information Loss of Discrete Tokens
Standard draft models operate strictly in token space. At each drafting step, the continuous hidden activations of the draft model are projected into discrete vocabulary tokens.
Collapsing continuous vector representations into discrete integer tokens discards nuanced contextual semantics. The draft model cannot condition its subsequent predictions on the rich, multi-layer semantic features captured by the target model’s upper transformer blocks.
3. The Failure of Static Multi-Head Drafting (Medusa)
Multi-head drafting frameworks (such as Medusa) attempt to eliminate separate draft models by adding parallel linear heads to the target LLM’s final layer. However, because these heads predict future tokens independently without conditioning on intermediate tokens, they violate the chain rule of probability ($P(x_{t+1}, x_{t+2} \mid x_{\le t}) \ne P(x_{t+1} \mid x_{\le t}) P(x_{t+2} \mid x_{\le t})$), causing grammatical incoherence and brittle acceptance rates.
The Solution: EAGLE-2 and Feature-Level Autoregression
Developed by researchers at SafeAI Lab and Peking University (Li et al., ICML 2024 / EMNLP 2024), EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) and its successor EAGLE-2 fundamentally resolve these limitations by shifting the drafting process from token space to feature space.
Rather than predicting discrete tokens with an external model, EAGLE performs autoregression over the second-to-top hidden state vectors (features) of the target model itself.
The Three Architectural Innovations of EAGLE-2
- Feature-Level Autoregression: A single, lightweight transformer decoder layer (the Auto-regression Head) models the sequence of feature vectors $\mathbf{f}_t \in \mathbb{R}^d$. Because $\mathbf{f}_t$ contains the synthesized knowledge of all 32–80 layers of the target model, drafting in feature space preserves semantic richness and achieves an acceptance rate of 80% to 90%.
- Re-use of the Target LM Head: The drafted feature vector $\mathbf{f}{t+1}$ is projected into token logits using the target model’s own original LM Head ($\mathbf{W}{\text{LM}}$). This completely eliminates vocabulary distribution shift.
- Context-Aware Dynamic Draft Trees (EAGLE-2): Instead of generating a fixed static draft tree, EAGLE-2 evaluates the contextual confidence of each drafted token. If confidence is high, it extends a deep linear draft chain; if confidence is low, it branches out into a wider tree, maximizing the expected number of accepted tokens per verification cycle.
Key Architectural Mechanics
1. The Recurrent Feature Drafting Step
Let $\mathbf{f}t \in \mathbb{R}^d$ denote the top-layer hidden state produced by the target LLM for token $y_t$. To predict the next feature vector $\mathbf{f}{t+1}$, the EAGLE Auto-regression Head combines the previous feature with the token embedding of the current candidate:
$$\mathbf{h}t^{\text{in}} = \mathbf{W}{\text{fuse}} \left[ \mathbf{f}_t , ; , \text{Emb}(y_t) \right]$$
The fused representation is passed through a single transformer decoder block equipped with causal self-attention:
$$\mathbf{f}_{t+1} = \text{TransformerLayer}\left(\mathbf{h}_t^{\text{in}}\right)$$
The candidate token distribution for step $t+1$ is then generated by passing $\mathbf{f}_{t+1}$ through the target model’s original linear output projection:
$$P(y_{t+1} \mid y_{\le t}) = \text{Softmax}\left( \mathbf{W}{\text{LM}} \mathbf{f}{t+1} \right)$$
This recurrent loop can be executed multiple times in microseconds to draft a sequence of future features and tokens: $\mathbf{f}{t+1} \to \mathbf{f}{t+2} \to \mathbf{f}_{t+3} \dots$
2. Context-Aware Dynamic Tree Construction in EAGLE-2
Text generation entropy varies drastically across natural language and code. Generating predictable syntax (e.g., for (int i = 0; i < N; i++)) has near-zero entropy, while creative writing exhibits high entropy with multiple valid continuations.
EAGLE-2 dynamically shapes the draft tree using a confidence score based on the ratio between the top two token probabilities:
$$C(y) = \frac{P(y_{(1)})}{P(y_{(2)})}$$
- High Confidence ($C(y) > \tau$): The drafting engine allocates its token budget to depth, constructing a long linear chain (up to 7 tokens deep).
- Low Confidence ($C(y) \le \tau$): The drafting engine allocates its token budget to breadth, generating 2 to 4 parallel branches at that node.
This dynamic tree pruning ensures that GPU verification resources are never wasted on speculative paths that have low probability of acceptance.
3. Single-Pass Tree Verification
The target LLM verifies the entire dynamic draft tree in a single batched forward pass using a specialized Tree Attention Mask:
- Tokens along the same branch can attend to their ancestral tokens.
- Tokens on separate branches cannot attend to each other, preventing causal cross-contamination.
The target model computes exact logits for every node in the tree simultaneously. A greedy or nucleus-sampling verifier traverses the tree from root to leaf, accepting matching tokens until the first discrepancy is reached.
Because verification is mathematically performed by the target LLM, the final generated text is 100% identical to vanilla autoregressive decoding.
Speculative Decoding Architecture Comparison
| Architectural Dimension | Classical Small Draft Model | Medusa Parallel Heads | DeepSeek-V3 MTP | EAGLE-2 Dynamic Tree |
|---|---|---|---|---|
| Drafting Domain | Discrete Token Space | Discrete Token Space | Embedded Feature Space | Top-Layer Feature Space |
| Draft Model Size | 1B – 7B standalone LLM | Small linear heads | Extra transformer layers | Single transformer decoder layer (~0.2B) |
| Distribution Shift | High (Independent training) | Moderate | Minimal (Trained joint) | Zero (Re-uses target LM Head) |
| Draft Tree Structure | Static chain | Static fixed tree | Linear depth | Dynamic context-aware tree |
| Token Acceptance Rate | 50% – 60% | 45% – 55% | 80% – 85% | 82% – 92% |
| Real-World Speedup | 1.4x – 1.8x | 1.3x – 1.6x | 1.8x – 2.2x | 3.0x – 4.5x |
| Quality Guarantee | Lossless (Exact match) | Lossless | Lossless | Lossless (Exact match) |
Reference Implementation: EAGLE Feature Auto-regression Head
Below is a self-contained PyTorch implementation demonstrating the EAGLE Auto-regression Head, feature-token concatenation, and dynamic confidence evaluation.
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple, List
class EAGLEAutoRegressionHead(nn.Module):
"""
Lightweight EAGLE Auto-regression Head:
Executes in feature space to draft next-token representations.
"""
def __init__(self, dim: int = 4096, vocab_size: int = 32000):
super().__init__()
self.dim = dim
self.vocab_size = vocab_size
# 1. Feature-Token Fusion Layer
self.fusion = nn.Linear(dim * 2, dim, bias=False)
self.norm_in = nn.RMSNorm(dim)
# 2. Single Transformer Decoder Layer
self.decoder_layer = nn.TransformerEncoderLayer(
d_model=dim,
nhead=16,
dim_feedforward=dim * 4,
activation="silu",
batch_first=True,
norm_first=True
)
def forward_step(
self,
f_t: torch.Tensor, # [1, 1, dim] Feature from previous step
emb_token: torch.Tensor, # [1, 1, dim] Embedding of current token
target_lm_head: nn.Linear # Target model's original LM head (frozen)
) -> Tuple[torch.Tensor, int, float]:
"""
Drafts the next feature vector f_{t+1}, samples candidate token, and evaluates confidence.
"""
# Step A: Fuse feature vector with token embedding
fused = torch.cat([f_t, emb_token], dim=-1)
x = self.fusion(fused)
x = self.norm_in(x)
# Step B: Transformer decoder extrapolation
f_next = self.decoder_layer(x) # [1, 1, dim] Next feature vector
# Step C: Project through target model's original LM head
with torch.no_grad():
logits = target_lm_head(f_next).squeeze(0).squeeze(0) # [vocab_size]
probs = F.softmax(logits, dim=-1)
# Extract top-2 candidates for confidence estimation
top2_probs, top2_indices = torch.topk(probs, k=2)
candidate_token = top2_indices[0].item()
# Confidence metric: ratio of top-1 to top-2 probability
confidence = (top2_probs[0] / (top2_probs[1] + 1e-8)).item()
return f_next, candidate_token, confidence
# Demonstration: Simulating Dynamic Tree Drafting
if __name__ == "__main__":
torch.manual_seed(42)
dim, vocab = 1024, 8000
# Instantiate EAGLE head and mock target model components
eagle_head = EAGLEAutoRegressionHead(dim=dim, vocab_size=vocab)
target_lm_head = nn.Linear(dim, vocab, bias=False)
token_embed = nn.Embedding(vocab, dim)
# Initial state: Top-layer hidden feature emitted by target LLM
initial_feature = torch.randn(1, 1, dim)
current_token = 42 # Starting token ID
print("--- Simulating EAGLE-2 Dynamic Feature Drafting ---")
current_f = initial_feature
drafted_sequence = [current_token]
for step in range(1, 6):
emb = token_embed(torch.tensor([[current_token]]))
next_f, next_token, confidence = eagle_head.forward_step(current_f, emb, target_lm_head)
# Decision: Deepen chain vs. branch based on confidence threshold (tau = 2.0)
action = "Deepen Chain" if confidence > 2.0 else "Branch Tree"
print(f"Draft Step {step} | Candidate Token: {next_token} | Confidence: {confidence:.2f} | Strategy: {action}")
drafted_sequence.append(next_token)
current_f = next_f
current_token = next_token
print(f"\nDrafted Candidate Sequence: {drafted_sequence}")
print("Ready for single-pass verification in target model!") Production Serving and Deployment Impacts
EAGLE-2 has been widely adopted across enterprise LLM infrastructure and open-source serving runtimes:
1. Native Integration in vLLM and SGLang
Modern serving engines provide first-class support for EAGLE and EAGLE-2 checkpoints:
# Launch vLLM with EAGLE speculative acceleration
vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
--speculative-model yuhuili/EAGLE-LLaMA3-Instruct-70B \
--num-speculative-tokens 5 \
--speculative-draft-tensor-parallel-size 1 Because the EAGLE head consists of only a single transformer layer (typically 0.2B to 0.4B parameters), its memory footprint is negligible compared to a 70B parameter base model, leaving virtually all GPU HBM allocated to the KV cache pool for high batch concurrency.
2. True Lossless Acceleration for High-Reliability Pipelines
Unlike heuristic prompt truncation or speculative lossy pruning, EAGLE-2 provides a strict mathematical guarantee: every emitted token is verified by the target foundation model.
For financial extraction, medical summarization, and mission-critical code generation, engineering teams achieve 3.0x to 4.5x latency reductions without compromising safety, accuracy, or reasoning integrity.
3. Accelerated Desktop AI Workflows in aiFetchly
For software engineers and data analysts using AiFetchly, EAGLE-2 transforms local workstation capabilities. By enabling EAGLE acceleration in local Ollama or vLLM backends, desktop agents generate complex code scripts, extract structured market tables, and conduct multi-step web intelligence sweeps at speeds exceeding 100 tokens per second on local GPU hardware—matching the responsiveness of cloud APIs while keeping sensitive data completely private.