speculative-decodingeagle-3llm-inferenceinference-accelerationtraining-time-testfeature-fusiontransformer-optimization

EAGLE-3 Guide: Scaling Speculative Decoding via TTT

October 9, 2026

EAGLE-3 Architecture: Scaling Speculative Decoding via Training-Time Test, Multi-Layer Feature Fusion, and Pure Classification Loss

The explosive growth of foundation models in enterprise production has placed inference throughput and token-generation latency at the center of artificial intelligence systems engineering. In autonomous multi-agent environments, complex financial data extraction engines, and interactive coding assistants powered by AI Fetchly, self-attention and autoregressive token decoding dominate infrastructure expenditures.

Over the past two years, Speculative Decoding (SD) has emerged as the industry standard for accelerating LLM generation without degrading output quality. Rather than relying on separate smaller foundation models that frequently desynchronize from the target model’s vocabulary, the EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) series pioneered feature-level speculative decoding. By training a lightweight single-layer transformer head directly on top of the target model’s intermediate representations, EAGLE-1 and EAGLE-2 established benchmark speedup records.

However, as foundation models scaled beyond 70 billion parameters, the EAGLE framework encountered a fundamental structural limitation: the feature prediction loss trap. In EAGLE-1 and EAGLE-2, the single-layer draft head was burdened with a dual objective: predicting upcoming token IDs while simultaneously reconstructing the dense 4,096-dimensional continuous hidden state vectors of the target model’s final layer. Forcing a lightweight draft head to solve complex high-dimensional regression compromised its primary goal—maximizing discrete token prediction accuracy.

To eliminate this bottleneck and push speculative acceleration to new heights, researchers from SafeAILab introduced EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (arXiv:2503.01840, accepted at NeurIPS 2025). EAGLE-3 introduces three transformative architectural breakthroughs: completely removing the feature prediction loss, introducing multi-layer feature fusion across early and deep transformer layers, and developing Training-Time Test (TTT) to simulate multi-step rollout errors directly during training.

The result is a dramatic acceleration leap: EAGLE-3 achieves up to 6.5x wall-clock speedups over autoregressive decoding (and 1.4x–1.8x over EAGLE-2) across LLaMA-3, Qwen-2.5, and DeepSeek model families, supported natively in production engines like vLLM and SGLang. In this comprehensive technical guide, we analyze the limitations of early feature speculation, examine the mathematical foundations of EAGLE-3, construct a functional PyTorch implementation, and evaluate production benchmarks.

The Feature Prediction Loss Trap in EAGLE-1 and EAGLE-2

To understand why EAGLE-3 required a fundamental training overhaul, we must examine the loss mechanics and information flow of prior feature-based speculative decoders.

The Dual-Objective Burden of Early EAGLE

In EAGLE-1 and EAGLE-2, the draft model consisted of a single transformer decoder layer appended to the target model’s final layer $N$. The draft head was trained using a composite loss function combining token classification with continuous feature regression:

$$\mathcal{L}{\text{EAGLE-1/2}} = \mathcal{L}{\text{cls}} + \lambda \cdot \mathcal{L}_{\text{feat}}$$

where $\mathcal{L}{\text{cls}}$ is standard cross-entropy over vocabulary tokens, and $\mathcal{L}{\text{feat}}$ is Smooth L1 loss penalizing the distance between the predicted feature vector $\hat{f}{t+1}$ and the target model’s true final-layer hidden state $f{t+1}^{(N)}$:

$$\mathcal{L}{\text{feat}} = \text{SmoothL1}\left(\hat{f}{t+1}, f_{t+1}^{(N)}\right)$$

While feature regression was originally hypothesized to stabilize multi-step drafting, it introduced three severe systemic drawbacks:

Gradient Conflict and Capacity Starvation: A single transformer layer has limited parameter capacity. Forcing it to reconstruct complex 4,096-dimensional continuous manifolds pulled optimization gradients away from discrete token classification, reducing the accuracy of emitted candidate tokens.
Exposure Bias during Multi-Step Rollouts: During training, the draft head was fed ground-truth target features at every step. During inference, however, the draft head must feed its own imperfectly predicted features into subsequent steps. This mismatch caused compounding error drift across speculative tree depth.
Single-Layer Information Bottleneck: By conditioning exclusively on the target model’s final hidden state $f^{(N)}$, the draft head ignored early and intermediate layer representations that encode essential lexical, morphological, and syntactic constraints.

The Evolution from EAGLE-2 to EAGLE-3:
┌────────────────────────────────────────────────────────────────────────┐
│ EAGLE-2: Dual Loss & Single-Layer Conditioning                        │
│ Final Layer N Features ──► Draft Head ──► Loss: L_cls + L_feat (L1)   │
│ ❌ Feature regression starves capacity; single-layer context choke.    │
├────────────────────────────────────────────────────────────────────────┤
│ EAGLE-3: Multi-Layer Fusion & Pure Classification (This Work)         │
│ Layers {2, N//2, N-3} ──► Multi-Layer Fusion ──► Draft Head ──► L_cls  │
│ ✔ Feature loss completely removed; Training-Time Test eliminates bias! │
└────────────────────────────────────────────────────────────────────────┘

When low-latency enterprise copilots orchestrate complex tool calls via data fetching APIs, draft-model latency directly compounds Time-to-First-Token (TTFT) and Inter-Token Latency (ITL). Maximizing draft acceptance accuracy is essential for production efficiency.

EAGLE-3 Architectural Blueprint: Three Core Breakthroughs

EAGLE-3 resolves these limitations through three foundational architectural and methodological innovations: Multi-Layer Feature Fusion, Complete Removal of Feature Loss, and Training-Time Test (TTT).

EAGLE-3 Three-Pillar Architecture:
┌────────────────────────────────────────────────────────────────────────┐
│ 1. MULTI-LAYER FEATURE FUSION: Concatenates {Layer 2, N//2, N-3}       │
│    Captures low-level syntax, mid-level entities, and deep semantics   │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ 2. REMOVAL OF FEATURE LOSS: Pure cross-entropy classification          │
│    L_total = L_cls (Discards L_feat entirely, maximizing token top-1)  │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ 3. TRAINING-TIME TEST (TTT): Simulates autoregressive inference rollout│
│    during training, exposing the draft head to its own output errors   │
└────────────────────────────────────────────────────────────────────────┘

1. Multi-Layer Feature Fusion

Rather than relying solely on the final hidden layer, EAGLE-3 extracts features from three strategically chosen intermediate layers of the target model with $N$ total layers:

$$\mathcal{L}_{\text{selected}} = \left{ 2, ; \lfloor N / 2 \rfloor, ; N - 3 \right}$$

Early Layer ($l=2$): Encodes low-level surface tokens, tokenization boundaries, and lexical syntax.
Middle Layer ($l=\lfloor N/2 \rfloor$): Encodes intermediate relational dependencies and entity bindings.
Penultimate Layer ($l=N-3$): Encodes abstract semantic reasoning before final output projection.

These three representations are projected and fused via a lightweight linear projection before being fed into the draft decoder layer:

$$f_{\text{fused}} = W_{\text{fuse}} \left[ f^{(2)} ; \Vert ; f^{(\lfloor N/2 \rfloor)} ; \Vert ; f^{(N-3)} \right]$$

This rich multi-tier representation provides the draft head with comprehensive syntactic and semantic context, dramatically improving its speculative predictions.

2. Complete Removal of Feature Prediction Loss

EAGLE-3 completely discards the continuous feature reconstruction loss $\mathcal{L}_{\text{feat}}$. The draft head is trained strictly on token-level cross-entropy loss:

$$\mathcal{L}{\text{EAGLE-3}} = -\sum{i=1}^k \log P_D\left(x_{t+i} \mid f_{\text{fused}}, \hat{x}_{<t+i}\right)$$

By relieving the draft model from the burden of reconstructing 4,096-dimensional vectors, 100% of the parameter capacity is dedicated to identifying the correct next-token distribution, boosting acceptance lengths by over 25%.

3. Training-Time Test (TTT)

To conquer exposure bias without expensive reinforcement learning, EAGLE-3 introduces Training-Time Test (TTT). During training, instead of feeding teacher-forced ground truth features into multi-step drafting, the model executes actual multi-step autoregressive decoding rollouts. By exposing the draft head to its own imperfect predictions during the training forward pass, the model learns to self-correct feature drift, ensuring robust long-horizon tree speculation.

Generational Comparison: EAGLE Evolution (EAGLE-1 to EAGLE-3)

The table below contrasts the architectural evolution across all three generations of the EAGLE framework:

Metric / Feature	EAGLE-1 (ICML 2024)	EAGLE-2 (EMNLP 2024)	EAGLE-3 (NeurIPS 2025)
Input Features	Target Final Layer $N$	Target Final Layer $N$	Multi-Layer Fusion ${2, N/2, N-3}$
Loss Function	$\mathcal{L}{\text{cls}} + \mathcal{L}{\text{feat}}$	$\mathcal{L}{\text{cls}} + \mathcal{L}{\text{feat}}$	Pure $\mathcal{L}_{\text{cls}}$ (Zero Feature Loss)
Rollout Robustness	Teacher-Forced (Biased)	Context-Aware Dynamic Tree	Training-Time Test (TTT)
Draft Head Scale	1 Transformer Layer	1 Transformer Layer	1 Transformer Layer (Expanded Capacity)
Average Speedup (vs. AR)	2.1x - 2.8x	2.5x - 3.4x	3.3x - 6.5x (Record Acceleration!)
Multi-Round Thinking Scaling	Moderate	Good	Superior (Maintains High Acceptance)

EAGLE-3 nearly doubles the acceleration ratio of early speculative decoders while preserving exact mathematical equivalence to the target model.

End-to-End Implementation: Building an EAGLE-3 Multi-Layer Drafter

Let us implement a complete, functional PyTorch module that encapsulates the EAGLE-3 multi-layer feature fusion, single-layer draft transformer, and classification loss computation.

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import List, Tuple, Dict

class EAGLE3DraftHead(nn.Module):
    """
    EAGLE-3 Multi-Layer Feature Fusion Draft Model.
    Fuses target model layers {2, N//2, N-3} and predicts speculative tokens
    using pure classification loss (zero feature regression).
    """
    def __init__(
        self,
        hidden_dim: int = 4096,
        draft_dim: int = 1024,
        vocab_size: int = 32000,
        num_fused_layers: int = 3
    ):
        super().__init__()
        self.hidden_dim = hidden_dim
        self.draft_dim = draft_dim
        self.vocab_size = vocab_size

        # 1. Multi-Layer Feature Fusion Projection
        # Concatenates hidden states from 3 selected layers: [Batch, SeqLen, 3 * HiddenDim]
        self.fuse_proj = nn.Linear(hidden_dim * num_fused_layers, draft_dim, bias=False)
        self.norm = nn.RMSNorm(draft_dim)

        # 2. Single-Layer Transformer Decoder Drafter
        decoder_layer = nn.TransformerDecoderLayer(
            d_model=draft_dim,
            nhead=8,
            dim_feedforward=draft_dim * 4,
            activation="gelu",
            batch_first=True
        )
        self.draft_decoder = nn.TransformerDecoder(decoder_layer, num_layers=1)

        # 3. Pure Token Classification Head
        self.lm_head = nn.Linear(draft_dim, vocab_size, bias=False)

    def forward(
        self,
        target_layers_hidden: List[torch.Tensor],
        candidate_embeds: torch.Tensor
    ) -> torch.Tensor:
        """
        Forward pass of EAGLE-3 draft model.
        Args:
            target_layers_hidden: List of 3 tensors [Batch, SeqLen, HiddenDim] from layers {2, N//2, N-3}.
            candidate_embeds: [Batch, DraftLen, DraftDim] embeddings of candidate tokens.
        Returns:
            Logits over vocabulary: [Batch, DraftLen, VocabSize].
        """
        assert len(target_layers_hidden) == 3, 'Must provide exactly 3 fused layer hidden states'

        # 1. Concatenate multi-layer hidden states along channel dimension
        concat_features = torch.cat(target_layers_hidden, dim=-1) # [Batch, SeqLen, 3 * HiddenDim]

        # 2. Project and normalize fused context representation
        memory_context = self.norm(self.fuse_proj(concat_features)) # [Batch, SeqLen, DraftDim]

        # 3. Autoregressive draft decoding with causal cross-attention
        draft_out = self.draft_decoder(tgt=candidate_embeds, memory=memory_context)

        # 4. Emit discrete token logits
        logits = self.lm_head(draft_out) # [Batch, DraftLen, VocabSize]
        return logits

    def compute_loss(
        self,
        logits: torch.Tensor,
        target_tokens: torch.Tensor
    ) -> torch.Tensor:
        """
        EAGLE-3 Loss: Pure Cross-Entropy Classification (Zero Feature Loss).
        """
        b, s, v = logits.shape
        loss = F.cross_entropy(logits.view(-1, v), target_tokens.view(-1))
        return loss

Functional Validation Harness

Let us verify the forward pass and classification loss computation:

def verify_eagle3_execution():
    device = 'cuda' if torch.cuda.is_available() else 'cpu'
    batch_size = 2
    seq_len = 32
    draft_len = 4
    hidden_dim = 4096
    draft_dim = 1024
    vocab_size = 32000

    print('Initializing EAGLE-3 Multi-Layer Draft Head...')
    model = EAGLE3DraftHead(hidden_dim=hidden_dim, draft_dim=draft_dim, vocab_size=vocab_size).to(device)

    # Mock 3 layers of hidden states from target model {Layer 2, N//2, N-3}
    h_early = torch.randn(batch_size, seq_len, hidden_dim, device=device)
    h_mid = torch.randn(batch_size, seq_len, hidden_dim, device=device)
    h_late = torch.randn(batch_size, seq_len, hidden_dim, device=device)
    target_hidden_list = [h_early, h_mid, h_late]

    cand_embeds = torch.randn(batch_size, draft_len, draft_dim, device=device)
    target_tokens = torch.randint(0, vocab_size, (batch_size, draft_len), device=device)

    # Forward and loss computation
    logits = model(target_hidden_list, cand_embeds)
    loss = model.compute_loss(logits, target_tokens)
    loss.backward()

    print(f"Logits Shape: {logits.shape}")
    print(f"Pure Classification Loss: {loss.item():.4f}")
    assert logits.shape == (batch_size, draft_len, vocab_size)
    assert model.fuse_proj.weight.grad is not None
    print('Verification Succeeded: EAGLE-3 multi-layer fusion and pure classification confirmed!')

if __name__ == '__main__':
    verify_eagle3_execution()

When building multi-step reasoning copilots that coordinate with custom backend endpoints, deploying an aligned speculative drafting head cuts API latency in half while ensuring zero mathematical drift.

Production Benchmarks & Empirical Acceleration

In rigorous benchmark evaluations published in NeurIPS 2025 (arXiv:2503.01840), EAGLE-3 was evaluated across LLaMA-3-8B/70B, Qwen-2.5-7B/72B, and DeepSeek-V3 on standard evaluation suites: MT-Bench, GSM8K, HumanEval, and CodeSearchNet.

Wall-Clock Generation Speedup

The table below reports measured end-to-end wall-clock speedup ratios across model sizes and tasks on NVIDIA A100 and H100 GPUs:

Target Model	Baseline Speed	EAGLE-1 Speedup	EAGLE-2 Speedup	EAGLE-3 Speedup	Average Acceptance ($\tau$)
LLaMA-3-8B-Instruct	1.00x	2.14x	2.58x	3.82x	3.91 tokens
Qwen-2.5-7B-Instruct	1.00x	2.21x	2.65x	4.10x	4.15 tokens
LLaMA-3-70B-Instruct	1.00x	2.45x	3.12x	4.85x - 6.50x	4.68 tokens
DeepSeek-V3 (671B MoE)	1.00x	2.30x	2.85x	4.40x	4.32 tokens

Wall-Clock Speedup on LLaMA-3-70B Serving:
Autoregressive (AR): [█████              ] 1.00x Baseline
EAGLE-1:             [██████████         ] 2.45x
EAGLE-2:             [█████████████      ] 3.12x
EAGLE-3:             [███████████████████] 5.25x - 6.50x (Record Speedup!)

EAGLE-3 consistently achieves over 3.8x speedup on 8B models and up to 6.5x on 70B models, maintaining an average acceptance length of $\tau > 4.5$ tokens per round.

Organizations evaluating infrastructure cost reductions and cloud compute allocations can explore flexible pricing structures to model cluster TCO savings when deploying high-throughput speculative decoding.

Production Deployment Guide: Serving EAGLE-3 with SGLang and vLLM

EAGLE-3 is open-source and officially supported in high-throughput inference engines like SGLang and vLLM:

# 1. Install SGLang with speculative decoding support
pip install "sglang[all]>=0.4.0"

# 2. Launch high-throughput server with EAGLE-3 drafter
python3 -m sglang.launch_server \
    --model-path meta-llama/Meta-Llama-3-70B-Instruct \
    --speculative-algorithm eagle3 \
    --speculative-draft yuhuili/EAGLE3-LLaMA3-70B-Instruct \
    --speculative-num-steps 5 \
    --tp-size 4 \
    --port 30000

Engineering teams can consult the full developer documentation for detailed configuration manifests, CUDA graph compilation flags, and multi-node Tensor Parallelism parameters.

Key Takeaways and Architectural Summary

Removing the Regression Trap: Completely discarding continuous feature prediction loss ($\mathcal{L}_{\text{feat}}$) focuses 100% of draft model capacity on discrete token prediction.
Multi-Layer Feature Fusion: Fuses intermediate representations from layers ${2, N/2, N-3}$, providing rich syntactic, relational, and semantic grounding.
Training-Time Test (TTT): Simulates multi-step autoregressive decoding errors during training, eliminating exposure bias.
Up to 6.5x Speedup: Slashes end-to-end latency by up to 6.5x over autoregressive decoding, outperforming EAGLE-2 across all benchmark suites.
Exact Lossless Fidelity: Retains mathematically exact output parity with the target model, guaranteeing zero perplexity drift.

EAGLE-3 cements speculative decoding as a mature, highly scalable systems technology, delivering enterprise-grade LLM acceleration for next-generation AI infrastructure.