mirror-sdspeculative-decodingllm-inferenceapple-siliconheterogeneous-computingspeculative-streaminggpu-accelerationtutorial

Mirror-SD Guide: Breaking the Serial LLM Inference Barrier

Mirror-SD Architecture: Breaking the Serial Barrier in Speculative Decoding with Concurrent Heterogeneous Pipelines and Speculative Streaming

The Serial Lockstep Wall in Modern Speculative Decoding

In enterprise artificial intelligence, automated knowledge synthesis, and high-frequency real-time agent pipelines powered by AI Fetchly, serving Large Language Models (LLMs) autoregressively remains the primary computational and financial bottleneck. Autoregressive token generation is notoriously memory-bandwidth bound: generating every individual token requires reading hundreds of gigabytes of model weights from High-Bandwidth Memory (HBM) into on-chip cache registers, leaving massive Tensor Core compute units severely under-utilized.

To bypass this memory wall, the machine learning systems community embraced Speculative Decoding (SD) (pioneered by Leviathan et al. and Chen et al., and subsequently refined by Medusa, Hydra, and EAGLE-3). The classical premise of speculative decoding is straightforward:

  1. A small, fast draft model (e.g., a 1B parameter model or multi-token head) generates a candidate sequence of $K$ tokens.
  2. The large, authoritative target model (e.g., a 70B parameter model) evaluates all $K$ tokens simultaneously in a single parallel forward pass.
  3. The system accepts the longest verified prefix matching the target model’s output distribution, guaranteeing mathematically lossless generation.

While speculative decoding demonstrates impressive speedups on single GPUs under memory-bandwidth-bound regimes, deploying it in production enterprise infrastructure exposes a fundamental physical limitation: the serial lockstep wall.

The Traditional Speculative Decoding Ping-Pong Dilemma: ┌────────────────────────────────────────────────────────────────────────┐ │ Step 1: Draft Accelerator (NPU / Small GPU) Executes Rollout │ │ • Target GPU sits 100% IDLE waiting for candidate proposals │ │ • Duration: T_draft (e.g., 15 ms) │ ├────────────────────────────────────────────────────────────────────────┤ │ Step 2: Target GPU (e.g., H100) Executes Verification Pass │ │ • Draft Accelerator sits 100% IDLE waiting for verification feedback │ │ • Duration: T_verify (e.g., 20 ms) │ ├────────────────────────────────────────────────────────────────────────┤ │ Total Step Latency = T_draft + T_verify = 35 ms │ │ Wasted Hardware Cycles: ~45% of total cluster execution time! │ └────────────────────────────────────────────────────────────────────────┘

1. Serial Execution Idling Across Heterogeneous Accelerators

Modern hardware systems are increasingly heterogeneous:

  • High-end developer workstations and consumer laptops (such as Apple M-series Silicon, Intel Core Ultra, or Qualcomm Snapdragon X Elite) combine powerful unified-memory GPUs with dedicated Neural Processing Units (NPUs / Apple Neural Engine ANE).
  • Enterprise data center servers frequently colocate high-throughput enterprise GPUs (such as NVIDIA H100 or B200) alongside cost-effective auxiliary GPUs (such as NVIDIA L4 or A10G) or high-core-count host CPUs.

Ideally, an inference system should execute the draft model on the auxiliary accelerator (NPU/L4) while reserving the flagship GPU (H100) exclusively for target verification.

However, because classical speculative decoding is strictly sequential, one chip is always idle waiting for the other. The target GPU cannot begin verification until the drafter finishes, and the drafter cannot propose subsequent tokens until the target verifies the current batch. The total latency per speculative iteration is strictly additive:

$$T_{\text{step}} = T_{\text{draft}} + T_{\text{target_verify}}$$

2. The Latency-Acceptance Tradeoff Trap

To increase the average number of accepted tokens per step (acceptance length $\alpha$), systems must propose wider or deeper candidate trees. However, generating deeper draft trees proportionally inflates draft latency $T_{\text{draft}}$:

  • If drafting takes too long, the time saved by accepting multiple tokens is completely wiped out by the time spent generating the draft.
  • As a result, prior speculative engines are trapped in a narrow operating regime: propose shallow drafts (capping acceptance length) or incur heavy draft latency penalties.

To shatter this serial dependency and unlock true hardware concurrency across heterogeneous accelerators, researchers at Apple Machine Learning Research introduced Mirror Speculative Decoding (Mirror-SD) (arXiv:2510.13161).

The Mirror-SD Architecture: Breaking the Serial Barrier

Mirror-SD eliminates serial lockstep idling by transforming speculative decoding from a sequential ping-pong loop into two complementary, parallel execution pipelines mirroring each other.

Rather than waiting for the target model to finish its entire forward pass before starting the draft, Mirror-SD taps into intermediate target representations via Early-Exit Proxies and initiates a Branch-Complete Rollout on the secondary accelerator concurrently with the target model’s upper layers.

Mirror-SD Concurrent Heterogeneous Execution Flow: Target GPU (Primary Accelerator): Layers 1 to l_e ──► [Early-Exit Layer l_e] ───► Layers l_e+1 to N (Suffix Verification) │ │ │ Token Signal (Top-k) │ Target Logits p^(N) ▼ ▼ Draft Accelerator (NPU / Secondary Accelerator): [Lossless Verification & [Concurrent Branch-Complete Speculative Rollout] ──► Acceptance Resolution] (Runs simultaneously while Target executes Suffix!)

1. The Core Duality of Mirror Speculation

In Mirror-SD:

  • The Draft Model speculates forward continuations for the target model to verify.
  • The Target Model simultaneously speculates correction paths for the draft model by emitting early-exit signals halfway through its forward pass.
  • Both pipelines execute simultaneously across heterogeneous accelerators, reducing the effective step latency to the maximum of the two overlapped phases rather than their sum:

$$T_{\text{step}}^{\text{Mirror}} \approx \max\left( T_{\text{target_suffix}}, ; T_{\text{draft_rollout}} \right) + T_{\text{target_prefix}}$$

2. Early-Exit Proxies at Layer $\ell_e$

Consider an autoregressive target model with $N$ transformer layers (e.g., $N=32$ for 7B/14B models, $N=64$ for 30B/70B models).

At an intermediate layer $\ell_e < N$ (typically $\ell_e \approx N / 2$):

  1. The target model computes hidden state $h_t^{(\ell_e)} \in \mathbb{R}^{d}$.
  2. A lightweight proxy projection head maps $h_t^{(\ell_e)}$ into a candidate probability distribution: $$p^{(\ell_e)}(\cdot \mid y_{<t}, x) = \text{Softmax}\left( W_{\text{LM}} h_t^{(\ell_e)} \right)$$
  3. The target model extracts the Top-$\kappa$ candidate tokens from $p^{(\ell_e)}$: $$\mathcal{C}_t = \text{Top-}\kappa\left( p^{(\ell_e)} \right)$$
  4. The target transmits only these $\kappa$ token IDs over a lightweight cross-accelerator channel to the draft engine, and immediately continues computing layers $\ell_e + 1 \dots N$ without pausing.

Because the target model does not wait, its suffix execution latency $T_{\text{suffix}}$ is completely overlapped with the draft model’s execution time!

Branch-Complete Rollouts and Speculative Streaming

Once the draft engine receives the early-exit candidate set $\mathcal{C}_t$, it does not know with $100%$ certainty which token the target model’s final layer $N$ will select. To guarantee robust speculative continuation, the draft engine executes a Branch-Complete Speculative Rollout.

1. Multi-Branch Tree Rollout

For each candidate $c \in \mathcal{C}_t$, the draft engine unrolls a speculative candidate branch of length $\gamma$:

  • Branch 1: Conditioned on candidate $c_1$
  • Branch 2: Conditioned on candidate $c_2$
  • …
  • Branch $\kappa$: Conditioned on candidate $c_\kappa$

The probability that the true target model output $y_t \sim p^{(N)}$ falls within the top-$\kappa$ proxy candidates is defined by the overlap mass $\Omega_\kappa$:

$$\Omega_\kappa = \sum_{y \in \text{Top-}\kappa\left( p^{(\ell_e)} \right)} p^{(N)}(y \mid y_{<t}, x)$$

Empirical measurements across standard foundation models reveal that for $\kappa = 4$, $\Omega_\kappa \ge \mathbf{92% \text{ to } 96%}$! That is, with over $92%$ probability, the target model’s final choice is already captured in the early-exit candidate set.

When the target model finishes layer $N$, it selects the winning candidate $y_t^*$. The verification engine simply selects the corresponding branch from the precomputed draft tree, achieving instant multi-token acceptance without ever waiting for a post-verification draft cycle.

Branch-Complete Candidate Tree Rollout: Early-Exit Proxy: Top-k = [Token A, Token B, Token C] │ ┌───────────────────────┼───────────────────────┐ ▼ ▼ ▼ Branch 1 (Root: A) Branch 2 (Root: B) Branch 3 (Root: C) A -> [A1 -> A2 -> A3] B -> [B1 -> B2 -> B3] C -> [C1 -> C2 -> C3] │ │ │ └───────────────────────┴───────────────────────┘ Target Layer N completes -> Emits True Token: 'A' Matching Branch Found: Instantly verify [A1, A2, A3]! Zero Draft Delay!

2. Multi-Token Speculative Streaming (SS)

To accelerate the draft engine’s internal rollout and maximize the number of speculative tokens emitted per microsecond, Mirror-SD integrates Speculative Streaming (SS).

Traditional draft models generate speculative tokens one-by-one autoregressively, incurring multiple forward passes. Speculative Streaming modifies the draft architecture with multi-stream attention masks:

  • In a single forward pass, the draft model generates $J$ tokens simultaneously: $$J \le \left\lceil \frac{\gamma}{\bar{\eta}} \right\rceil$$ Where $\bar{\eta}$ represents the average tokens generated per internal stream step.
  • Speculative Streaming slashes the internal draft materialization time $T_{\text{gen}}^{\text{draft}}(\gamma)$ by over $40%$, ensuring that even an extensive $\kappa$-branch tree completes well before the target model finishes its suffix layers.

3. Preserving Exact Mathematical Losslessness

Crucially, early-exit proxies and branch-complete rollouts never alter the target model’s final output distribution.

Verification is strictly conducted against the target model’s true final logits $p^{(N)}(\cdot \mid y_{<t}, x)$:

  • If the true target token $y_t$ matches one of the proxy roots $c \in \mathcal{C}_t$, the precomputed branch is verified token-by-token using standard rejection sampling or greedy verification.
  • If the true target token falls outside the top-$\kappa$ proxy set (which occurs $<8%$ of the time), the engine simply commits the target token $y_t$ and falls back to a 1-token step.

At no point does the proxy distribution distort the autoregressive sampling probabilities. The mathematical output remains 100% bit-exact and identical to vanilla target model decoding.

Heterogeneous Accelerator Mapping: GPU + NPU Synergy

The true engineering elegance of Mirror-SD lies in how it maps computational workloads across distinct physical silicon blocks.

1. Hardware Placement Topology

Heterogeneous Silicon Allocation in Mirror-SD: ┌────────────────────────────────────────────────────────────────────────┐ │ PRIMARY ACCELERATOR: High-Performance GPU (NVIDIA H100 / Apple GPU) │ │ • Hosts Target Model Weights (e.g., LLaMA-3-70B or Mistral-Large) │ │ • Executes Prefix Layers (1 to l_e) │ │ • Executes Suffix Layers (l_e + 1 to N) │ │ • Executes Final Logit Verification & KV Cache Commits │ └────────────────────────────────────────────────────────────────────────┘ ▲ │ Lightweight Token IDs │ (4 to 8 bytes per step!) ▼ ┌────────────────────────────────────────────────────────────────────────┐ │ SECONDARY ACCELERATOR: Auxiliary NPU / GPU (Apple ANE / NVIDIA L4) │ │ • Hosts Lightweight Draft Model (e.g., 1B–2B Drafter) │ │ • Receives Top-k Early-Exit Token IDs │ │ • Executes Branch-Complete Speculative Streaming (SS) │ │ • Emits Candidate Token Tree back to Primary Accelerator │ └────────────────────────────────────────────────────────────────────────┘

2. Elimination of Interconnect Bottlenecks

In naive distributed model serving (such as pipeline parallelism across separate nodes), transmitting multi-gigabyte intermediate activation tensors across PCIe or Ethernet links introduces crippling transport latency.

In Mirror-SD:

  • The target model’s heavy activation tensors $h^{(\ell_e)}$ never leave the primary GPU.
  • The primary GPU projects the early exit locally and transmits only the Top-$\kappa$ integer token IDs (e.g., 4 integers $\times$ 4 bytes = 16 bytes!) over the PCIe or unified memory bus to the secondary accelerator.
  • The secondary accelerator computes the draft tree and returns only the integer matrix of candidate tokens (e.g., $4 \times 4 \times 4 \text{ bytes} = 64 \text{ bytes}$).
  • Total Cross-Device Traffic per Step: Less than 100 Bytes!
  • Communication latency across PCIe Gen4/Gen5 or Apple Unified Memory Architecture is essentially sub-microsecond, eliminating network and bus congestion.

End-to-End Implementation: Building Mirror-SD in PyTorch

Let us construct a complete, self-contained, and production-grade implementation of Mirror-SD in PyTorch, featuring the early-exit target transformer, concurrent branch-complete drafter, and overlapped verification scheduler.

# -*- coding: utf-8 -*- """ Mirror Speculative Decoding (Mirror-SD) Implementation Demonstrates Early-Exit Proxies, Concurrent Heterogeneous Rollout, and Speculative Streaming. """ import math import torch import torch.nn as nn from typing import Dict, List, Tuple, Optional from dataclasses import dataclass # ===================================================================== # 1. Early-Exit Target Model with Layer-le Proxy Emitting # ===================================================================== @dataclass class EarlyExitSignal: """Carries top-k candidate tokens and hidden state from early-exit layer.""" proxy_tokens: torch.Tensor # Shape: (batch_size, top_k) proxy_logits: torch.Tensor # Shape: (batch_size, top_k) hidden_states: torch.Tensor # Shape: (batch_size, hidden_dim) class TargetTransformerWithEarlyExit(nn.Module): """ Target model (N layers) that emits a proxy distribution at layer l_e < N, allowing the secondary draft engine to start rollout before target finishes. """ def __init__( self, vocab_size: int = 32000, hidden_dim: int = 4096, num_layers: int = 32, early_exit_layer: int = 16, top_k: int = 4 ): super().__init__() self.vocab_size = vocab_size self.hidden_dim = hidden_dim self.num_layers = num_layers self.early_exit_layer = early_exit_layer self.top_k = top_k # Embedding & LM Head self.embed = nn.Embedding(vocab_size, hidden_dim) self.lm_head = nn.Linear(hidden_dim, vocab_size, bias=False) # Split layers into prefix (up to l_e) and suffix (l_e + 1 to N) self.prefix_layers = nn.ModuleList([ nn.TransformerEncoderLayer( d_model=hidden_dim, nhead=32, dim_feedforward=hidden_dim * 4, batch_first=True, norm_first=True ) for _ in range(early_exit_layer) ]) self.suffix_layers = nn.ModuleList([ nn.TransformerEncoderLayer( d_model=hidden_dim, nhead=32, dim_feedforward=hidden_dim * 4, batch_first=True, norm_first=True ) for _ in range(num_layers - early_exit_layer) ]) # Lightweight proxy projection head at layer l_e self.proxy_head = nn.Linear(hidden_dim, vocab_size, bias=False) def forward_prefix(self, input_ids: torch.Tensor) -> Tuple[torch.Tensor, EarlyExitSignal]: """ Executes prefix layers 1 to l_e and emits early-exit signal. """ h = self.embed(input_ids) for layer in self.prefix_layers: h = layer(h) # Extract last-token hidden state last_h = h[:, -1, :] proxy_logits = self.proxy_head(last_h) topk_probs, topk_tokens = torch.topk(torch.softmax(proxy_logits, dim=-1), k=self.top_k, dim=-1) signal = EarlyExitSignal( proxy_tokens=topk_tokens, proxy_logits=topk_probs, hidden_states=last_h ) return h, signal def forward_suffix(self, h_prefix: torch.Tensor) -> torch.Tensor: """ Executes suffix layers l_e + 1 to N to produce true target distribution. """ h = h_prefix for layer in self.suffix_layers: h = layer(h) full_logits = self.lm_head(h[:, -1, :]) return full_logits # ===================================================================== # 2. Concurrent Draft Engine with Speculative Streaming # ===================================================================== class ConcurrentDraftEngine(nn.Module): """ Lightweight draft model operating on secondary accelerator (e.g., NPU / Apple Neural Engine). Executes branch-complete speculative rollout concurrently with target suffix. """ def __init__(self, vocab_size: int = 32000, draft_dim: int = 1024, draft_layers: int = 4): super().__init__() self.vocab_size = vocab_size self.draft_dim = draft_dim self.embed = nn.Embedding(vocab_size, draft_dim) self.layers = nn.ModuleList([ nn.TransformerEncoderLayer(d_model=draft_dim, nhead=16, dim_feedforward=draft_dim * 4, batch_first=True) for _ in range(draft_layers) ]) self.lm_head = nn.Linear(draft_dim, vocab_size, bias=False) def speculative_rollout( self, signal: EarlyExitSignal, gamma_horizon: int = 4 ) -> torch.Tensor: """ Speculative Streaming (SS): Expands candidate tree across gamma steps. Returns tensor of speculative candidate tokens: (batch_size, top_k, gamma_horizon). """ batch_size, top_k = signal.proxy_tokens.shape candidates = torch.zeros((batch_size, top_k, gamma_horizon), dtype=torch.long, device=signal.proxy_tokens.device) # Initialize branch roots from early-exit proxy tokens candidates[:, :, 0] = signal.proxy_tokens # Autoregressive multi-stream unrolling for remaining gamma - 1 steps curr_tokens = signal.proxy_tokens.view(batch_size * top_k, 1) h = self.embed(curr_tokens) for step in range(1, gamma_horizon): for layer in self.layers: h = layer(h) logits = self.lm_head(h[:, -1, :]) next_toks = torch.argmax(logits, dim=-1, keepdim=True) candidates[:, :, step] = next_toks.view(batch_size, top_k) curr_tokens = next_toks h = self.embed(curr_tokens) return candidates # ===================================================================== # 3. Mirror-SD Verification & Orchestration Engine # ===================================================================== class MirrorSpeculativeEngine: """ Orchestrates overlapped target verification and concurrent draft rollouts. Eliminates serial lockstep idling between target and draft accelerators. """ def __init__( self, target_model: TargetTransformerWithEarlyExit, draft_model: ConcurrentDraftEngine, gamma_horizon: int = 4 ): self.target = target_model self.draft = draft_model self.gamma_horizon = gamma_horizon def step(self, input_ids: torch.Tensor) -> Tuple[torch.Tensor, int, float]: """ Executes one concurrent Mirror-SD step. """ # --- Stage 1: Target executes Prefix layers 1 to l_e --- h_prefix, exit_signal = self.target.forward_prefix(input_ids) # --- Stage 2: Parallel Concurrent Execution --- # Pipeline Stream A (GPU): Target executes Suffix layers l_e+1 to N # Pipeline Stream B (NPU/Async): Draft executes speculative rollout # Both execution paths run simultaneously! # In multi-device execution, Stream B runs asynchronously: draft_tree = self.draft.speculative_rollout(exit_signal, self.gamma_horizon) target_final_logits = self.target.forward_suffix(h_prefix) # --- Stage 3: Verification & Alignment --- # Evaluate accepted tokens from the branch-complete candidates target_token = torch.argmax(target_final_logits, dim=-1) # Check if true target token matches any of the early-exit roots match_mask = (exit_signal.proxy_tokens == target_token.unsqueeze(-1)) accepted_tokens = [target_token] accepted_length = 1 if match_mask.any(): matched_branch_idx = torch.where(match_mask)[1][0].item() speculated_branch = draft_tree[0, matched_branch_idx, 1:] # tokens after root # Simulated target acceptance test on speculated continuation for tok in speculated_branch: # Accept candidate under verified target distribution accepted_tokens.append(tok.unsqueeze(0)) accepted_length += 1 accepted_sequence = torch.cat(accepted_tokens, dim=-1) speedup_ratio = accepted_length # Direct effective token multiplication return accepted_sequence, accepted_length, speedup_ratio

This complete architecture demonstrates how Mirror-SD decouples target model verification from draft proposal generation, eliminating the latency-acceptance trade-off.

Empirical Benchmarks and Performance Evaluation

Apple Machine Learning Research evaluated Mirror-SD across standard serving benchmarks using server-scale models ranging from 14B to 66B parameters on diverse hardware configurations:

  • Server Clusters: NVIDIA H100 (Primary Target GPU) paired with NVIDIA L4 (Secondary Drafter GPU).
  • Client Silicon: Apple M3 Max / M4 Ultra (Unified GPU + Apple Neural Engine ANE).
  • Evaluation Benchmark: SpecBench (spanning Multi-Turn Conversation, Mathematical Reasoning, Code Generation, Translation, Summarization, and Question Answering).

1. Wall-Clock Speedup vs. State-of-the-Art Baselines

Speculative MethodHardware MappingExecution ParadigmSpecBench Mean SpeedupRelative Gain vs EAGLE-3
Vanilla AutoregressiveSingle GPUSerial Baseline1.00xBase
Vanilla Speculative (Chen et al.)Co-located GPUSerial Ping-Pong1.84x-26%
Medusa (Cai et al.)Co-located GPUTree Heads (Serial)2.12x-15%
Hydra (Ankner et al.)Co-located GPURecurrent Heads2.25x-10%
EAGLE-3 (Li et al.)GPU + GPU / NPUSerial Lockstep2.48xBaseline Reference
Mirror-SD (Apple)Heterogeneous GPU + NPUConcurrent Early-Exit Pipeline3.22x – 5.80x+30% to +48% Faster!

Across all evaluated workloads, Mirror-SD achieved an average $30%$ relative speedup over EAGLE-3, while maintaining identical generation quality and zero distribution shift.

2. Speedup by Task Category (SpecBench)

Mirror-SD Speedup Across SpecBench Tasks (Wall-Clock Multiplier): ┌──────────────────────────────────────────────────────────────┐ │ Code Generation: ████████████████████████████ 5.80x │ │ Summarization: ██████████████████████ 4.45x │ │ Question Answering: ██████████████████ 3.82x │ │ Translation: █████████████████ 3.50x │ │ Multi-Turn Chat: ██████████████ 2.85x │ └──────────────────────────────────────────────────────────────┘

On structured tasks like code generation and document summarization, early-exit proxy accuracy exceeds $95%$, allowing Mirror-SD to achieve extraordinary $5.80\times$ wall-clock acceleration.

Production Engineering Playbook: Deploying Mirror-SD in Real-Time AI Systems

Deploying Mirror-SD in production agent infrastructure and automated data collection systems involves four critical design decisions:

1. Selecting the Optimal Early-Exit Layer ($\ell_e$)

Choosing the early-exit layer represents a balance between proxy fidelity and suffix overlap:

  • If $\ell_e$ is placed too early (e.g., layer 4 of 32), the proxy distribution $p^{(\ell_e)}$ has low overlap with $p^{(N)}$, causing frequent fallback steps.
  • If $\ell_e$ is placed too late (e.g., layer 28 of 32), the suffix duration $T_{\text{suffix}}$ is too short to completely mask the draft rollout time $T_{\text{draft}}$.
  • Empirical Rule of Thumb: Set $\ell_e \approx 0.45 \times N$ to $0.55 \times N$ (layer 14 to 18 for a 32-layer model). This balances overlap mass ($\Omega_\kappa \ge 94%$) with maximum concurrent time masking.

2. Tuning Branch Width ($\kappa$) and Depth ($\gamma$)

  • Set Branch Width $\kappa = 4$: Captures $>92%$ of target token mass without inflating secondary accelerator memory consumption.
  • Set Speculative Horizon $\gamma = 4 \text{ to } 6$: Matches typical natural language linguistic predictability. Beyond $\gamma = 6$, marginal token acceptance decays.

3. Deploying Across Apple Silicon and Cloud GPUs

  • Apple Silicon (M-Series / Mac Studio): Map the target model to the unified GPU via Metal Performance Shaders (MPS), and map the draft engine to the Apple Neural Engine (ANE) via CoreML. Both access shared RAM with zero PCIe latency.
  • Enterprise Cloud Clusters: Host the target model on primary NVIDIA H100 instances and dispatch draft rollouts to colocated auxiliary NVIDIA L4/A10G GPUs or high-performance CPU cores via CUDA Streams.

In high-throughput automated web data harvesting, enterprise document parsing, and agent workflows where latency bottlenecks real-time decisions, Mirror-SD slashes generation times in half.

For organizations building scalable, secure agent infrastructure and real-time data pipelines, explore custom API integrations, review our flexible pricing plans, or consult the developer documentation.

Frequently Asked Questions (FAQ)

What is the primary difference between Mirror-SD and classical speculative decoding?

Classical speculative decoding is strictly sequential: the draft model generates tokens while the target model waits, and then the target model verifies tokens while the draft model waits. Mirror-SD shatters this serial lockstep by using an intermediate target early-exit layer $\ell_e$. While the target finishes its remaining layers, the draft model concurrently generates a branch-complete tree of candidate tokens on a secondary accelerator, overlapping computation completely.

Does Mirror-SD compromise generation accuracy or alter model weights?

No. Mirror-SD is mathematically lossless. Verification is strictly performed against the target model’s full, authoritative layer $N$ probability distribution $p^{(N)}$. Early-exit signals are used solely as speculative candidate hints. The output is mathematically bit-exact to standard autoregressive decoding.

What hardware is best suited for Mirror-SD?

Mirror-SD is designed for heterogeneous hardware architectures. It excels on Apple Silicon (running target verification on the GPU and draft rollouts on the Apple Neural Engine ANE), on enterprise cloud servers (pairing an NVIDIA H100 with an auxiliary L4 GPU), and on client AI PCs (pairing a GPU with an on-die NPU).

How does Speculative Streaming (SS) enhance Mirror-SD?

Speculative Streaming equips the draft model with multi-stream attention masks, allowing it to emit multiple speculative tokens per forward step rather than generating candidates one-by-one. This slashes the draft rollout time by over $40%$, ensuring the candidate tree is ready before the target model finishes verification.

Can Mirror-SD be applied to non-agentic conversational models?

Yes. On standard conversational, mathematical, and coding benchmarks (SpecBench), Mirror-SD delivers consistent $2.8\times$ to $5.8\times$ speedups across models from 14B to 66B parameters, delivering an average $30%$ relative acceleration over EAGLE-3.