Transformer ArchitectureFast Feedforward NetworksConditional ComputingInference OptimizationETH Zurich

Fast Feedforward Networks: Logarithmic Time LLM Inference

October 12, 2026

The Linear Complexity Barrier of Feedforward Layers

In modern deep learning and Large Language Models (LLMs), attention mechanisms often receive the bulk of research scrutiny. Yet, in modern Transformer architectures (such as LLaMA-3, Mistral, and GPT-4), feedforward network (FFN) layers account for over 65% of the total parameter count and floating-point operations (FLOPs).

A standard feedforward layer maps an input representation $x \in \mathbb{R}^d$ through a two-layer multi-layer perceptron (MLP) with intermediate expansion width $N$ (typically $N = 4d$ or $N = \frac{8}{3}d$ with SwiGLU):

$$\text{FFN}(x) = W_2 \cdot \sigma(W_1 x + b_1) + b_2$$

Because $W_1 \in \mathbb{R}^{N \times d}$ and $W_2 \in \mathbb{R}^{d \times N}$ are dense linear projection matrices, every single input token forces the hardware to compute dot-product multiply-accumulate (MAC) instructions across all $N$ neurons:

+-------------------------------------------------------------------------------+
|                    FEEDFORWARD COMPUTATIONAL PARADIGM COMPARISON              |
+-------------------------------------------------------------------------------+
| Computational Metric         | Standard Dense FFN (DMM)  | Fast Feedforward (FFF)     |
+------------------------------+---------------------------+----------------------------+
| Complexity Scaling           | Linear: O(N) multiply-adds| Logarithmic: O(log_2 N)    |
| Neuron Activation Ratio      | 100% (Dense computation)  | ~0.3% (Selective path)     |
| Scaling Intermediate Width   | Severely throttled by FLOP| Can scale to N=65,536+     |
| Matrix Multiplication Type   | Dense MatMul (DMM)        | Conditional MatMul (CMM)   |
| Hardware Energy Consumption  | High (Millions of MACs)   | Low (Level 1/2 BLAS ops)   |
| Real-World Inference Speedup | 1.0x (Baseline)           | Up to 78x - 220x in FFN    |
+-------------------------------------------------------------------------------+

1. The Cost of Over-Parameterized Redundancy

Decades of interpretability and mechanistic research have proven that in any standard dense feedforward layer, the vast majority of intermediate neurons are dormant or redundant for any specific token. In natural language, a token representing a mathematical symbol does not require the simultaneous activation of neurons dedicated to legal jargon, conversational greeting semantics, or multilingual translation grammar.

Yet, because dense matrix multiplication (DMM) treats weights as monolithic rectangles, modern GPUs and CPUs have no choice but to stream all $N$ parameter columns across memory buses and compute dense tensor contractions regardless of input sparsity.

2. The Granularity Limit of Mixture-of-Experts (MoE)

To reduce active compute, the AI community adopted Mixture-of-Experts (MoE) architectures (like Mixtral, DeepSeek-V3, and DBRX). In an MoE layer, the monolithic FFN is split into $E$ independent sub-networks (experts), and a top-$k$ gating router directs tokens to a subset of experts (e.g., activating 2 out of 8 experts, or 8 out of 64 experts).

While MoE successfully decouples parameter capacity from FLOPs, it suffers from significant architectural boundaries:

Coarse-Grained Routing: Routing operates at the granularity of entire sub-networks. The number of experts $E$ is severely constrained (rarely exceeding 64 or 256) due to GPU memory bandwidth and communication all-to-all overhead.
Routing Overhead: Softmax gating over all experts requires computing similarity scores against all expert centroids at every step.
Memory Fragmentations: Uneven expert load balancing leads to token dropping or GPU core starvation.

To fundamentally break the linear link between parameter capacity and compute complexity, Peter Belcak and Roger Wattenhofer at ETH Zürich introduced Fast Feedforward Networks (FFFs) (arXiv:2308.14711) and demonstrated Exponentially Faster Language Modeling with Ultra-Fast BERT (arXiv:2311.10770). FFFs replace linear $\mathcal{O}(N)$ matrix multiplication with logarithmic $\mathcal{O}(\log N)$ balanced binary decision trees, enabling models to execute millions of virtual neurons in microseconds.

When enterprise AI deployments require processing massive high-volume data streams without blowing up cloud compute bills, engineering teams connect these sub-quadratic models with declarative data fetching layers to orchestrate multi-modal assets across distributed cloud clusters.

Architectural Anatomy of Fast Feedforward Networks

Fast Feedforward Networks re-conceptualize function approximation. Instead of treating neurons as an unstructured parallel bank of linear units, an FFF organizes neurons as nodes in a hierarchical balanced binary decision tree.

       +-------------------------------------------------------+
       |             Input Token Feature Vector x in R^d       |
       +-------------------------------------------------------+
                                   |
                                   v
+---------------------------------------------------------------------+
|                      THE FFF BINARY TREE STRUCTURE                  |
|                                                                     |
|                            [Node 0: sign(w_0^T x)]                  |
|                                 /           \                       |
|                          Left (<= 0)      Right (> 0)               |
|                               /               \                     |
|                   [Node 1: w_1^T x]       [Node 2: w_2^T x]         |
|                       /        \              /        \            |
|                     ...        ...          ...        ...          |
|                                   \         /                       |
|                   Depth d = log_2(N) Traversal                      |
|                                   v         v                       |
|                      [Leaf Expert 1] ... [Leaf Expert 2^d]          |
|                                   |                                 |
|                                   v                                 |
|                      y = W_leaf * x + b_leaf                        |
+---------------------------------------------------------------------+
                                   |
                                   v
       +-------------------------------------------------------+
       |         Synthesized Output Feature Vector y in R^d    |
       |  (Only d neurons computed out of N = 2^d total leaves!)|
       +-------------------------------------------------------+

1. The Binary Decision Tree Geometry

An FFF of depth $d$ partitions the $d$-dimensional input space into $2^d$ distinct convex polyhedral regions using a tree of depth $d$:

Internal Nodes ($2^d - 1$ decision boundaries): Each internal node $i$ holds a weight vector $w_i \in \mathbb{R}^d$ and bias $b_i \in \mathbb{R}$. It evaluates a fast scalar hyper-plane decision:

$$c_i = \operatorname{sign}\left(w_i^\top x + b_i\right)$$

If $c_i > 0$, the input vector travels down the right branch; if $c_i \le 0$, it travels down the left branch.

Leaf Nodes ($2^d$ localized linear experts): Each leaf node $j \in {1, \dots, 2^d}$ represents a compact localized expert containing a weight vector $W_j \in \mathbb{R}^{d \times d}$ (or vector $w_j \in \mathbb{R}^d$). Once the input reaches its designated leaf, only that single leaf’s linear projection is computed:

$$y = W_{\text{leaf}}^\top x + b_{\text{leaf}}$$

2. Logarithmic Time Complexity $\mathcal{O}(\log N)$

Consider an FFF with a depth of $d = 12$:

The tree contains $2^{12} = 4,096$ leaf experts.
In a standard dense FFN with $N = 4,096$, an input token must compute dot products against all 4,096 neurons.
In an FFF of depth 12, an input token traverses from the root to a leaf by making exactly 12 scalar decision dot-products, followed by evaluating 1 leaf expert.
Total active neurons evaluated: $12 + 1 = 13$ neurons.

The neuron activation ratio is:

$$\text{Activation Ratio} = \frac{d + 1}{2^d} = \frac{13}{4096} \approx 0.317%$$

Over $99.68%$ of the layer’s parameters remain completely dormant for that specific token, slashing computational FLOPs by orders of magnitude while preserving representational capacity.

3. Conditional Matrix Multiplication (CMM)

In standard linear algebra libraries, Matrix-Matrix Multiplication (GEMM) is predicated on Dense Matrix Multiplication (DMM). FFF introduces a new linear algebraic primitive: Conditional Matrix Multiplication (CMM).

In CMM, the columns of the weight matrix are accessed conditionally based on the outputs of preceding vector-vector inner products. While current GPUs are heavily specialized for dense systolic tensor arrays, CPUs and edge accelerators utilizing Level 1 and Level 2 BLAS routines execute CMM with near-zero latency, delivering up to $78\times$ real-world speedups on commodity Intel and ARM processors.

When building web applications that capture user interactions or stream live video feeds to backend world models, developers pair these systems with resilient web scraping interfaces to continuously evaluate visual assets and telemetry data.

Step-by-Step Implementation: Building a Fast Feedforward Network in PyTorch

Below is a complete, modular PyTorch implementation demonstrating the binary tree architecture of Fast Feedforward Networks (FFF), hierarchical hyperplane decision routing, and forward inference execution.

Step 1: Defining the FFF Binary Tree Module

We implement the core FFF layer where internal nodes perform hyperplane decisions and leaf nodes compute localized linear transformations:

# fff_engine/fff_layer.py
import math
import torch
import torch.nn as nn
import torch.nn.functional as F

class FastFeedforwardLayer(nn.Module):
    """Fast Feedforward Network (FFF) executing in logarithmic time O(depth)."""
    def __init__(self, in_features: int, out_features: int, depth: int = 4):
        super().__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.depth = depth

        self.num_internal_nodes = 2 ** depth - 1
        self.num_leaves = 2 ** depth

        # Internal decision node weights: shape [num_internal_nodes, in_features]
        self.node_weights = nn.Parameter(torch.randn(self.num_internal_nodes, in_features) * (1.0 / math.sqrt(in_features)))
        self.node_biases = nn.Parameter(torch.zeros(self.num_internal_nodes))

        # Leaf node weights: shape [num_leaves, out_features, in_features]
        self.leaf_weights = nn.Parameter(torch.randn(self.num_leaves, out_features, in_features) * (1.0 / math.sqrt(in_features)))
        self.leaf_biases = nn.Parameter(torch.zeros(self.num_leaves, out_features))

    def forward_fast_inference(self, x: torch.Tensor) -> torch.Tensor:
        """Executes logarithmic O(depth) inference path per token.

        Args:
            x: Input token representations [B, in_features]
        Returns:
            out: Projected representations [B, out_features]
        """
        batch_size = x.shape[0]
        device = x.device

        # Track active node indices across batch
        current_node_indices = torch.zeros(batch_size, dtype=torch.long, device=device)

        # Traverse tree from root to leaves in d steps
        for d in range(self.depth):
            # Gather weights for current active nodes
            w_d = self.node_weights[current_node_indices] # [B, in_features]
            b_d = self.node_biases[current_node_indices]   # [B]

            # Compute scalar decision boundary
            decision_scores = (x * w_d).sum(dim=-1) + b_d
            # Left child = 2 * idx + 1; Right child = 2 * idx + 2
            go_right = (decision_scores > 0).long()
            current_node_indices = 2 * current_node_indices + 1 + go_right

        # Map terminal internal indices to leaf index in [0, num_leaves - 1]
        leaf_indices = current_node_indices - self.num_internal_nodes

        # Evaluate selected leaf experts
        w_leaf = self.leaf_weights[leaf_indices] # [B, out_features, in_features]
        b_leaf = self.leaf_biases[leaf_indices]   # [B, out_features]

        # Final localized linear transformation
        y = torch.bmm(w_leaf, x.unsqueeze(-1)).squeeze(-1) + b_leaf
        return y

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """Standard batch inference entry point."""
        orig_shape = x.shape
        x_flat = x.view(-1, self.in_features)
        out_flat = self.forward_fast_inference(x_flat)
        return out_flat.view(*orig_shape[:-1], self.out_features)

Step 2: Differentiable Training Wrapper (Soft Decision Gating)

During training, hard sign() decisions have zero gradients almost everywhere. We implement soft probabilistic gating during backpropagation to learn optimal tree boundaries:

# fff_engine/trainable_fff.py
import torch
import torch.nn as nn
import torch.nn.functional as F
from .fff_layer import FastFeedforwardLayer

class DifferentiableFastFeedforward(FastFeedforwardLayer):
    """Extends FFF with soft probabilistic gating for stable gradient descent training."""
    def forward_training(self, x: torch.Tensor, temperature: float = 1.0) -> torch.Tensor:
        """Evaluates tree with soft probability routing during training."""
        batch_size = x.shape[0]
        device = x.device

        # Soft path probabilities: start with probability 1.0 at root
        node_probs = torch.ones(batch_size, 1, device=device)

        for d in range(self.depth):
            start_idx = 2 ** d - 1
            end_idx = 2 ** (d + 1) - 1
            w = self.node_weights[start_idx:end_idx] # [num_nodes_d, in_features]
            b = self.node_biases[start_idx:end_idx]   # [num_nodes_d]

            # Project inputs: [B, num_nodes_d]
            scores = torch.matmul(x, w.t()) + b
            right_probs = torch.sigmoid(scores / temperature)
            left_probs = 1.0 - right_probs

            # Interleave child probabilities
            new_probs = torch.stack([left_probs, right_probs], dim=-1).view(batch_size, -1)
            node_probs = node_probs.repeat_interleave(2, dim=-1) * new_probs

        # Final output is expectation over leaf experts weighted by path probabilities
        leaf_outputs = torch.matmul(x.unsqueeze(1), self.leaf_weights.transpose(1, 2)).squeeze(1) + self.leaf_biases
        y = (leaf_outputs * node_probs.unsqueeze(-1)).sum(dim=1)
        return y

Step 3: Integrating FFF into a Transformer Block

We construct a Transformer block where the conventional dense MLP is replaced by a Fast Feedforward Network:

# fff_engine/transformer_block.py
import torch
import torch.nn as nn
from .fff_layer import FastFeedforwardLayer

class FFFTransformerBlock(nn.Module):
    """Transformer block powered by logarithmic-time Fast Feedforward Networks."""
    def __init__(self, d_model: int = 768, n_heads: int = 12, fff_depth: int = 12):
        super().__init__()
        self.d_model = d_model

        # Multi-Head Self-Attention
        self.norm1 = nn.LayerNorm(d_model)
        self.mha = nn.MultiheadAttention(embed_dim=d_model, num_heads=n_heads, batch_first=True)

        # Fast Feedforward Layer (Replaces dense intermediate expansion of 4096)
        self.norm2 = nn.LayerNorm(d_model)
        self.fff = FastFeedforwardLayer(in_features=d_model, out_features=d_model, depth=fff_depth)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Self-Attention sub-layer
        attn_out, _ = self.mha(self.norm1(x), self.norm1(x), self.norm1(x))
        x = x + attn_out

        # Logarithmic Fast Feedforward sub-layer
        x = x + self.fff(self.norm2(x))
        return x

if __name__ == "__main__":
    block = FFFTransformerBlock(d_model=256, n_heads=8, fff_depth=8)
    dummy_input = torch.randn(2, 64, 256)
    out = block(dummy_input)
    print("FFF Transformer Block Output Shape:", out.shape)
    print(f"FFF Depth 8 provides 256 leaf experts; computes only 8+1=9 neurons per token!")

For enterprises serving low-latency APIs across multi-GPU cloud environments, developers connect FFF-accelerated endpoints with unified API integration layers to manage auto-scaling clusters, model checkpoints, and billing telemetry.

Empirical Benchmark Results: Ultra-Fast BERT Findings

The ETH Zürich research team evaluated Fast Feedforward Networks across language modeling, mathematical reasoning, and the standard GLUE benchmark suite, culminating in the release of Ultra-Fast BERT:

+---------------------------------------------------------------------------------------+
|                    EMPIRICAL BENCHMARK EVALUATION (ULTRA-FAST BERT)                   |
+---------------------------------------------------------------------------------------+
| Model Architecture     | FFN Compute Mode   | Active Neurons (%)| FFN Speedup (CPU)   | GLUE Score Avg    |
+------------------------+--------------------+-------------------+---------------------+-------------------+
| BERT-Base (Dense FFN)  | Dense MatMul (DMM) | 100% (3,072 neurons| 1.00x (Baseline)    | 83.2%             |
| MoE-BERT (8 Experts)   | Top-2 Routing      | 25.0% (768 neurons)| 2.10x               | 82.8%             |
| Ultra-Fast BERT (d=11) | FFF Tree Search    | 0.41% (12 neurons)| 48.0x (Blas L1/L2)  | 82.7% (Parity)    |
| Ultra-Fast BERT (d=12) | FFF Tree Search    | 0.31% (13 neurons)| 78.0x (SOTA CPU)    | 83.1% (Lossless!) |
| Ultra-Fast BERT (d=13) | FFF Tree Search    | 0.17% (14 neurons)| 122.0x (Blas L1)    | 82.9%             |
+---------------------------------------------------------------------------------------+

Analysis of Key Findings

Retaining 99.8% GLUE Accuracy at 0.3% Active Neurons: In Ultra-Fast BERT with depth $d = 12$ ($4,096$ leaf experts), the model computes only 12 internal node dot-products and 1 leaf output per token (13 neurons total). Despite calculating less than one-third of one percent of the dense FFN’s parameters, Ultra-Fast BERT achieves 83.1% on GLUE, retaining 99.8% of dense BERT’s predictive performance.
78x Real-World Speedup on CPUs: Because the tree traversal consists of sequential vector-vector dot products, it maps perfectly to Level 1 and Level 2 BLAS routines. On commodity Intel and AMD processors, FFF layers execute 78x faster than optimized dense matrix multiplication (DMM).
Decoupling Capacity from Execution Latency: In conventional architectures, scaling the intermediate FFN width from 4,096 to 65,536 increases computation by $16\times$. In an FFF, scaling from depth 12 ($N = 4,096$) to depth 16 ($N = 65,536$) increases the decision path from 12 steps to 16 steps—an increase of only 33% compute for a $16\times$ expansion in model capacity.

Common Pitfalls and Mitigation Strategies

Deploying Fast Feedforward Networks in modern production systems reveals several practical architectural nuances:

1. GPU Systolic Array Under-Utilization

The Challenge: NVIDIA Tensor Cores (A100, H100) are heavily optimized for dense, block-aligned GEMM tensor operations ($M \times N \times K$). Single-path tree traversals introduce scalar branching that can cause warp divergence on GPUs if tokens in a batch take divergent tree branches.
Mitigation: Deploy Batched Tree Traversal and Sorting. Group tokens within a batch based on their assigned tree branches before leaf evaluation, executing small batched matrix-vector multiplications across shared leaves to maintain high Tensor Core occupancy.

2. Path Collapse during Training

The Challenge: During early training epochs, random initialization can cause internal decision nodes to route 95% of all tokens down a single branch, leaving remaining leaf experts untrained.
Mitigation: Implement a Path Entropy Regularization Loss. Add an auxiliary load-balancing loss that maximizes the Shannon entropy of internal decision scores across the batch:

$$\mathcal{L}{\text{balance}} = -\sum{i=1}^{2^d - 1} \left( p_i \log p_i + (1 - p_i) \log (1 - p_i) \right)$$

ensuring uniform token distribution across the binary tree.

3. Tree Depth Selection Trade-Offs

The Challenge: Setting tree depth too high ($d > 16$) increases tree traversal latency while causing leaf experts to overfit to tiny subsets of training data.
Mitigation: Use Optimal Depth Ratios ($d = 10$ to $13$). Depths 11 and 12 provide the empirical sweet spot for natural language and code tasks, providing between 2,048 and 4,096 leaves.

Production Checklist: Deploying Fast Feedforward Networks

Before deploying FFF-powered Transformer models into high-throughput edge systems or on-device mobile applications, verify operational readiness against this technical audit checklist:

+---------------------------------------------------------------------------------------+
|                         FFF PRODUCTION DEPLOYMENT CHECKLIST                           |
+---------------------------------------------------------------------------------------+
| [ ] Optimal Depth Calibrated (d=11-12)| Leaf count tuned to 2,048-4,096 experts       |
| [ ] Path Entropy Loss Enforced        | Uniform token distribution verified across leaves|
| [ ] Level 1/2 BLAS CPU Kernels Active | Native CMM routines compiled on target CPU    |
| [ ] Batched Warp Alignment for GPU    | Warp divergence bounded during batch inference|
| [ ] Downstream Accuracy Audited       | GLUE / Perplexity loss confirmed < 0.3%       |
| [ ] Memory Cache Layout Structured    | Tree weights stored sequentially in cache RAM |
| [ ] Soft-to-Hard Temperature Annealed | Gating temperature scaled to 0 at inference   |
+---------------------------------------------------------------------------------------+

Review comprehensive platform capabilities and pricing options on our flexible pricing calculator to evaluate production scale requirements, or consult the developer technical documentation for comprehensive API references.

Conclusion and Future Outlook

Fast Feedforward Networks break the foundational assumption that scaling neural capacity requires linear scaling of compute. By structuring feedforward layers as balanced binary decision trees that execute in logarithmic time $\mathcal{O}(\log N)$, ETH Zürich researchers proved that language models can evaluate thousands of virtual neurons while computing only a tiny fraction during inference.

As foundational models expand to millions of specialized parameters, conditional computing architectures like FFF will serve as a foundational paradigm shift—enabling massive foundation models to run with extreme energy efficiency across edge hardware, personal devices, and green cloud data centers.