jambamambamixture-of-expertstransformerllm-architecturetutorial

Jamba Guide: Hybrid Transformer-Mamba MoE Architecture

Jamba Architecture: Joint State-Space SSMs, Attention Anchors, 16-Expert MoE & 256k Context Serving

The Long-Context Trilemma: Transformers vs. Pure SSMs vs. MoE

In modern enterprise artificial intelligence, automated financial analysis, and large-scale web research platforms powered by AI Fetchly, serving long context windows (such as 128,000 to 256,000 tokens) has transitioned from an experimental research goal into a mission-critical production requirement. Whether processing multi-year SEC 10-K filings, auditing full software repositories, or synthesizing complex technical documentation harvested via real-time data fetching, foundation models must retain factual access across vast sequence histories.

However, as engineering teams scaled production architectures to hundreds of thousands of tokens, they confronted three seemingly irreconcilable physical constraints—The Long-Context Trilemma:

The Long-Context Architectural Trilemma: [ High In-Context Associative Recall ] ▲ / \ / \ / \ / \ [ Linear O(N) Compute ] ─────── [ Massive Capacity & Efficiency ] (Pure State-Space SSMs) (Sparse Mixture-of-Experts) The Core Trade-offs: 1. Standard Transformers (LLaMA, GPT-4): • High Associative Recall: Full all-to-all attention achieves 100% on needle retrieval. • Quadratic KV Cache Wall: Consumes 300+ GB of VRAM solely for KV states on 128k contexts. 2. Pure State-Space Models (Mamba, Mamba-2): • Linear O(N) Compute & Memory: Zero KV cache growth (O(1) hidden state). • Associative Recall Collapse: Fails on complex in-context retrieval and multi-hop reasoning. 3. Sparse Mixture-of-Experts (Mixtral 8x7B): • Massive Parameter Capacity: High capacity with low active FLOPs. • Inherits Quadratic Attention: Still suffers from linear KV cache blowup on long inputs.

The Limitations of Pure Attention and Pure Mamba

To understand why hybrid architectures are necessary, consider the mathematical failure modes of existing single-paradigm systems:

  1. The KV Cache Memory Exhaustion of Transformers: In pure Transformer decoders, each layer must retain Key and Value activations for every past token. For a 70B parameter model at 256k context length, the KV cache alone requires over 670 GB of GPU High-Bandwidth Memory (HBM). Even an 8-GPU node of NVIDIA H100 (80GB) cards runs completely out of memory before executing a single decode step.
  2. The Information Bottleneck of Pure Mamba: Selective State Space Models (Mamba / S6) compress historical context into a fixed-size latent state vector $h_t \in \mathbb{R}^{d_{\text{state}}}$. While this provides constant $O(1)$ memory per token and linear $O(N)$ training time, compressing 256,000 tokens into a finite hidden state vector inevitably triggers information loss. On challenging in-context learning benchmarks (such as complex phone-book lookups, multi-step code tracking, and long-range associative recall), pure Mamba models degrade significantly compared to attention-based transformers.

To dismantle this trilemma and achieve the optimal balance of near-zero KV cache growth, perfect long-context associative recall, and high parameter capacity, AI21 Labs introduced Jamba in 2024. Jamba is the world’s first production-scale hybrid architecture that combines Mamba state-space blocks, Transformer self-attention anchors, and Sparse Mixture-of-Experts (MoE) layers within a unified backbone.

The Jamba Architecture: Interleaving Attention, Mamba, and MoE

Jamba introduces a modular Jamba Block hierarchy that strategically interleaves different layer types to optimize both compute throughput and memory bandwidth.

The Jamba Layer Topology: Total Model Scale: 52 Billion Total Parameters / 12 Billion Active Parameters Context Window: 256,000 Tokens (Up to 140k on a single 80GB GPU) A Single Jamba Block (Repeated 4 Times for a total of 32 Layers): Layer 1: [ Mamba SSM Layer ] + [ Dense MLP ] Layer 2: [ Mamba SSM Layer ] + [ Sparse MoE (16 Experts, Top-2) ] Layer 3: [ Mamba SSM Layer ] + [ Dense MLP ] Layer 4: [ Mamba SSM Layer ] + [ Sparse MoE (16 Experts, Top-2) ] Layer 5: [ Mamba SSM Layer ] + [ Dense MLP ] Layer 6: [ Mamba SSM Layer ] + [ Sparse MoE (16 Experts, Top-2) ] Layer 7: [ Mamba SSM Layer ] + [ Dense MLP ] Layer 8: [ Self-Attention ] + [ Sparse MoE (16 Experts, Top-2) ] ◄── Attention Anchor! (1 Attention : 7 Mamba)

1. The 1:7 Layer Interleaving Ratio

In standard transformers, 100% of the layers use self-attention. In Jamba, only 1 out of every 8 layers uses self-attention, while the remaining 7 layers use Mamba selective state-space blocks.

This 1:7 interleaving ratio possesses critical mathematical properties:

  • 8x Smaller KV Cache Footprint: Because Key and Value caches are only allocated and stored for the attention layers, the total KV cache memory footprint is reduced by $8\times$ compared to a standard transformer. A 256k sequence that would normally consume 320 GB of KV cache in LLaMA-3 requires only 40 GB in Jamba!
  • Periodic Associative Anchoring: Mamba layers compress and process the high-throughput token stream linearly. Every 8th layer, a multi-head self-attention block acts as an associative anchor, scanning the global sequence to retrieve specific historical details and refreshing the Mamba representations.
  • Hardware-Friendly Grouped Query Attention (GQA): The attention layers utilize Grouped-Query Attention with 8 KV heads, further minimizing memory bandwidth saturation during generation.

2. Alternating Sparse Mixture-of-Experts (MoE)

In addition to hybrid sequence modeling, Jamba integrates Sparse Mixture-of-Experts (MoE) into its feed-forward network (FFN) layers:

  • Alternating MoE Placement: MoE is applied on every other layer (Layers 2, 4, 6, 8, etc.). The intermediate layers (Layers 1, 3, 5, 7) utilize standard dense MLPs.

  • 16 Specialized Experts: Each MoE layer contains 16 distinct expert networks.

  • Top-2 Router Mechanism: For every token, a learned gating router evaluates the token hidden state and routes it to the top 2 experts:

    $$y = \sum_{i \in \text{Top-2}(G(x))} G(x)_i \cdot \text{Expert}_i(x)$$

    where $G(x) = \text{softmax}(\text{Top-2}(W_g x))$.

  • Capacity without Compute Bloat: Jamba has 52 billion total parameters, but only 12 billion active parameters are used to process any single token during forward inference. This delivers the knowledge capacity of a 50B+ model with the inference speed and FLOP cost of a lightweight 12B model.

Inside the Mamba State-Space Layers: Mathematical Mechanics

To understand how Jamba achieves linear complexity across 7 out of 8 layers, we examine the Mamba Selective State Space (S6) engine.

Selective State Space Model (S6) Forward Flow: Continuous ODE: h'(t) = A h(t) + B x(t) y(t) = C h(t) Discretization via Step Size Δ: Ā_t = exp(Δ_t · A) B̄_t = Δ_t · B_t Recurrent State Update (Linear Time O(N)): h_t = Ā_t · h_{t-1} + B̄_t · x_t y_t = C_t · h_t Key Innovation: Input-Dependent Selectivity • B_t, C_t, and Δ_t are dynamically projected from input x_t! • The model decides which tokens to write into memory and which to forget.

1. Continuous to Discrete State Space Formulation

The continuous-time state space equation maps an input signal $x(t) \in \mathbb{R}$ to an output $y(t) \in \mathbb{R}$ via an internal hidden state vector $h(t) \in \mathbb{R}^{d_{\text{state}}}$:

$$h’(t) = A h(t) + B x(t)$$

$$y(t) = C h(t)$$

where $A \in \mathbb{R}^{d_{\text{state}} \times d_{\text{state}}}$ is structured as a diagonal or HiPPO matrix to model long-range memory decay.

In Mamba, the continuous system is discretized using a dynamic timescale parameter $\Delta_t$:

$$\bar{A}_t = \exp(\Delta_t A), \quad \bar{B}_t = (\Delta_t A)^{-1} (\exp(\Delta_t A) - I) \cdot \Delta_t B_t \approx \Delta_t B_t$$

The discrete recurrent update is computed as:

$$h_t = \bar{A}t h{t-1} + \bar{B}_t x_t$$

$$y_t = C_t h_t$$

2. Hardware-Aware Parallel Associative Scan

While the recurrent equation $h_t = \bar{A}t h{t-1} + \bar{B}_t x_t$ is sequential in nature, during training it can be computed across entire sequences in parallel using a Parallel Associative Scan.

Because state transitions satisfy associativity:

$$(h_t, \bar{A}t) \circ (h{t-1}, \bar{A}_{t-1}) = (\bar{A}t h{t-1} + \bar{B}_t x_t, \bar{A}t \bar{A}{t-1})$$

The prefix sum of state updates across sequence length $L$ is computed on GPU Tensor Cores using a tree reduction in $O(\log L)$ parallel steps, achieving full hardware saturation without sequential for-loops.

Furthermore, Mamba’s CUDA kernel keeps the hidden state $h_t$ resident in fast GPU SRAM (on-chip cache), avoiding slow round-trips to HBM and achieving 3x higher memory bandwidth efficiency than standard attention.

Mixture-of-Experts: Top-2 Gating and Load Balancing

In Jamba’s MoE layers, routing tokens efficiently across 16 experts without creating routing bottlenecks is crucial for distributed stability.

Jamba Top-2 Gating Mechanism: Token Embedding x ∈ ℝ^d │ ▼ [ Gating Router W_g ∈ ℝ^(16 × d) ] ──► Compute Router Logits: H(x) = W_g · x │ ▼ Select Top-2 Expert Indices: {e₁, e₂} = argtop2( H(x) ) │ ▼ Compute Normalized Routing Weights: P_i = exp(H(x)_i) / ( exp(H(x)_{e₁}) + exp(H(x)_{e₂}) ) │ ┌─────┴─────┐ ▼ ▼ [ Expert e₁ ] [ Expert e₂ ] (Selected from 16 SwiGLU Experts) │ │ └─────┬─────┘ ▼ Weighted Sum: y = P_{e₁} · Expert_{e₁}(x) + P_{e₂} · Expert_{e₂}(x)

1. Top-2 Gating Formulation

Given input representation $x \in \mathbb{R}^{d_{\text{model}}}$, the router computes unnormalized routing logits across all $E = 16$ experts:

$$H(x) = W_g x \in \mathbb{R}^{16}$$

The router selects the two highest-scoring experts:

$$\mathcal{K} = \text{Top-2}(H(x)) = {e_1, e_2}$$

The routing weights are normalized over the selected top-2 candidates via softmax:

$$P_i(x) = \frac{\exp(H(x)i)}{\sum{j \in \mathcal{K}} \exp(H(x)_j)}, \quad i \in \mathcal{K}$$

Each expert is an independent two-layer MLP with SwiGLU activation:

$$\text{Expert}i(x) = \left( \text{Swish}(x W{1, i}) \otimes (x W_{2, i}) \right) W_{3, i}$$

2. Auxiliary Load Balancing Loss ($\mathcal{L}_{\text{balance}}$)

Without regularization, gating routers suffer from expert collapse: the router routes 90% of tokens to the same 2 or 3 popular experts, leaving the remaining experts untrained and creating severe memory imbalances across GPU workers.

To enforce uniform expert utilization, Jamba optimizes an Auxiliary Load Balancing Loss:

$$\mathcal{L}{\text{balance}} = \alpha \cdot E \sum{i=1}^E f_i \cdot P_i$$

where:

  • $E = 16$ is the total number of experts.
  • $f_i = \frac{1}{T} \sum_{t=1}^T \mathbb{I}(\text{Expert } i \in \text{Top-2}(x_t))$ is the fraction of tokens routed to expert $i$.
  • $P_i = \frac{1}{T} \sum_{t=1}^T P_i(x_t)$ is the average routing probability assigned to expert $i$.
  • $\alpha = 0.01$ is the auxiliary balancing weight.

When expert distribution is perfectly uniform ($f_i = \frac{2}{E}$ and $P_i = \frac{2}{E}$ for all $i$), the loss reaches its theoretical minimum, preventing GPU communication stalls in distributed Tensor Parallelism.

Hands-On Implementation: Building a Hybrid Jamba Block in PyTorch

In this section, we implement a production-grade, functional PyTorch module representing the core Hybrid Jamba Block—integrating Mamba state-space recurrence, self-attention anchoring, and Top-2 Sparse MoE routing.

1. PyTorch Implementation of the Hybrid Jamba Layer Stack

import math import torch import torch.nn as nn import torch.nn.functional as F from typing import List, Tuple, Optional class SimpleMambaSSM(nn.Module): """ Simulated Mamba Selective State Space (S6) core in pure PyTorch. Computes linear O(N) sequence recurrence with input-dependent discretization. """ def __init__(self, d_model: int, d_state: int = 16): super().__init__() self.d_model = d_model self.d_state = d_state # Log-scale transition matrix A: (d_model, d_state) self.A_log = nn.Parameter(torch.log(torch.arange(1, d_state + 1, dtype=torch.float32).repeat(d_model, 1))) # Projections for dynamic B, C, and Δ self.proj_ssm = nn.Linear(d_model, d_state * 2 + 1, bias=False) self.dt_proj = nn.Linear(1, d_model, bias=True) def forward(self, x: torch.Tensor) -> torch.Tensor: B, L, D = x.shape N = self.d_state # Dynamic projections proj = self.proj_ssm(x) B_t = proj[:, :, :N] C_t = proj[:, :, N:2*N] dt_param = proj[:, :, 2*N:] delta = F.softplus(self.dt_proj(dt_param)) A = -torch.exp(self.A_log) delta_A = torch.exp(delta.unsqueeze(-1) * A.unsqueeze(0).unsqueeze(0)) delta_B = delta.unsqueeze(-1) * B_t.unsqueeze(2) # Sequential scan (in production, replaced by fused CUDA associative scan) h = torch.zeros(B, D, N, device=x.device, dtype=x.dtype) y_list = [] for t in range(L): x_t = x[:, t, :].unsqueeze(-1) h = delta_A[:, t] * h + delta_B[:, t] * x_t C_step = C_t[:, t, :].unsqueeze(1) y_t = torch.sum(h * C_step, dim=-1) y_list.append(y_t) return torch.stack(y_list, dim=1) class SparseMoELayer(nn.Module): """ Top-2 Sparse Mixture-of-Experts Layer with 16 SwiGLU Experts. """ def __init__(self, d_model: int, d_ff: int, num_experts: int = 16, top_k: int = 2): super().__init__() self.num_experts = num_experts self.top_k = top_k self.router = nn.Linear(d_model, num_experts, bias=False) # List of SwiGLU expert MLPs self.experts = nn.ModuleList([ nn.Sequential( nn.Linear(d_model, d_ff, bias=False), nn.SiLU(), nn.Linear(d_ff, d_model, bias=False) ) for _ in range(num_experts) ]) def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: B, L, D = x.shape flat_x = x.reshape(-1, D) # (N_tokens, D) # 1. Router logits logits = self.router(flat_x) # (N_tokens, num_experts) topk_weights, topk_indices = torch.topk(logits, self.top_k, dim=-1) topk_weights = F.softmax(topk_weights, dim=-1) # 2. Dispatch tokens to selected experts flat_out = torch.zeros_like(flat_x) for k in range(self.top_k): expert_idx = topk_indices[:, k] weights = topk_weights[:, k].unsqueeze(-1) for exp_id in range(self.num_experts): mask = (expert_idx == exp_id) if mask.any(): tokens = flat_x[mask] exp_out = self.experts[exp_id](tokens) flat_out[mask] += weights[mask] * exp_out # Compute simple load balance loss freq = torch.bincount(topk_indices.reshape(-1), minlength=self.num_experts).float() / (flat_x.shape[0] * self.top_k) avg_prob = F.softmax(logits, dim=-1).mean(dim=0) aux_loss = self.num_experts * torch.sum(freq * avg_prob) return flat_out.reshape(B, L, D), aux_loss class JambaHybridBlock(nn.Module): """ Modular Jamba 8-Layer Hybrid Block implementing the 1:7 Ratio: - 7 Mamba SSM layers - 1 Self-Attention layer - Alternating Sparse MoE FFN layers """ def __init__(self, d_model: int = 256, d_ff: int = 512, num_heads: int = 4): super().__init__() self.layers = nn.ModuleList() for i in range(8): is_attention = (i == 7) # Layer 8 is Attention Anchor is_moe = (i % 2 == 1) # Layers 2, 4, 6, 8 are MoE # Sequence mixer (Mamba or Multi-Head Attention) if is_attention: mixer = nn.MultiheadAttention(d_model, num_heads, batch_first=True) else: mixer = SimpleMambaSSM(d_model, d_state=16) # Feed-forward mixer (Dense MLP or Sparse MoE) if is_moe: ffn = SparseMoELayer(d_model, d_ff, num_experts=8, top_k=2) else: ffn = nn.Sequential( nn.Linear(d_model, d_ff), nn.GELU(), nn.Linear(d_ff, d_model) ) self.layers.append(nn.ModuleDict({ "is_attn": is_attention, "is_moe": is_moe, "norm1": nn.LayerNorm(d_model), "mixer": mixer, "norm2": nn.LayerNorm(d_model), "ffn": ffn })) def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: total_aux_loss = 0.0 for layer in self.layers: # 1. Sequence Mixer res = x norm_x = layer["norm1"](x) if layer["is_attn"]: mixer_out, _ = layer["mixer"](norm_x, norm_x, norm_x) else: mixer_out = layer["mixer"](norm_x) x = res + mixer_out # 2. FFN / MoE Mixer res = x norm_x = layer["norm2"](x) if layer["is_moe"]: ffn_out, aux_l = layer["ffn"](norm_x) total_aux_loss = total_aux_loss + aux_l else: ffn_out = layer["ffn"](norm_x) x = res + ffn_out return x, total_aux_loss # Verification Unit Test if __name__ == "__main__": batch_size = 2 seq_len = 32 d_model = 128 d_ff = 256 jamba_block = JambaHybridBlock(d_model=d_model, d_ff=d_ff, num_heads=4) x_in = torch.randn(batch_size, seq_len, d_model) out, loss = jamba_block(x_in) print("Jamba Hybrid 8-Layer Block Output Shape:", out.shape) print(f"Auxiliary MoE Balance Loss: {loss.item():.4f}") assert out.shape == (batch_size, seq_len, d_model) print("Jamba Hybrid Block Architecture Validated Successfully!")

2. Loading Jamba with Hugging Face Transformers

AI21 Labs released official weights for Jamba (ai21labs/Jamba-v0.1 and ai21labs/AI21-Jamba-1.5-Mini):

import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "ai21labs/AI21-Jamba-1.5-Mini" # 1. Load Tokenizer and Hybrid Model tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto", attn_implementation="flash_attention_2" ) # 2. Evaluate Massive Long-Context Input (Up to 256k Tokens) long_document_prompt = "Document analysis prompt: " + ("The quick brown fox jumps over the lazy dog. " * 5000) inputs = tokenizer(long_document_prompt, return_tensors="pt").to("cuda") # 3. Autoregressive Generation with 8x Smaller KV Cache with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=256, temperature=0.7, top_p=0.9, use_cache=True ) response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True) print("Jamba Generation Output:\n", response)

When building automated web scraping and document extraction pipelines with our automated web scraping tools, passing full scraped multi-page domains directly into Jamba models via our custom API integration enables comprehensive cross-document summarization without context fragmentation.

Empirical Benchmarks & Performance Analysis

To evaluate Jamba’s architectural capabilities, AI21 Labs benchmarked it across general academic capabilities (MMLU, GSM8k, HumanEval), long-context retrieval (Needle-in-a-Haystack across 256k tokens), and serving throughput on NVIDIA A100/H100 hardware.

1. General Benchmark Evaluation

The following table summarizes performance comparing LLaMA-2-70B, Mixtral 8x7B, Command R+, and Jamba (52B total / 12B active):

Model ArchitectureTotal ParamsActive ParamsMMLU (5-shot)GSM8k (CoT)HumanEval (pass@1)Maximum Context Length
LLaMA-2-70B70B (Dense)70B68.9%56.8%32.3%4,096 tokens
Mixtral 8x7B47B (MoE)13B70.6%74.4%40.2%32,768 tokens
LLaMA-3-8B8B (Dense)8B68.4%79.6%62.2%8,192 tokens
Command R+104B (Dense)104B75.7%72.8%53.7%128,000 tokens
Jamba-v0.152B (MoE+SSM)12B67.4%59.9%37.8%256,000 tokens
Jamba 1.5 Mini52B (MoE+SSM)12B74.1%83.2%64.5%256,000 tokens (SOTA)

2. 256k Needle-in-a-Haystack Retrieval

A primary concern with non-transformer sequence models is the degradation of long-range associative recall. On the Needle-in-a-Haystack benchmark evaluated up to 256,000 tokens:

  • Pure Mamba models typically show green scores up to 16k tokens, but degrade to orange/red (failure) beyond 64k tokens.
  • Jamba achieved a perfect 100% green retrieval score across all 256k depths and sequence lengths. The periodic 1:7 self-attention anchors prevent state decay, providing the exact associative retrieval power of full-attention transformers.

3. Serving Throughput and VRAM Footprint

In production serving comparisons evaluated on an 80GB NVIDIA A100 GPU:

  • Maximum Context on a Single 80GB GPU:
    • LLaMA-2-70B (FP16): Crashes (OOM) beyond 32,000 tokens.
    • Mixtral 8x7B (FP16): Crashes beyond 32,000 tokens.
    • Jamba: Easily fits a 140,000-token context on a single 80GB GPU without quantization or offloading!
  • Inference Throughput on Long Contexts (128k Tokens): Jamba delivers $3.0\times$ higher token generation throughput compared to Mixtral 8x7B, transforming long-document AI from an expensive luxury into an economically viable workload.

Production Engineering: Serving & Hardware Best Practices

Deploying Jamba in enterprise environments requires understanding hardware topology, batching parameters, and inference framework configuration.

High-Throughput Enterprise Jamba Topology: ┌─────────────────────────┐ ┌───────────────────────────────┐ │ Client Query / 128k Doc │ ──► │ Dynamic Context Ingestion │ └─────────────────────────┘ └──────────────┬────────────────┘ │ ▼ ┌───────────────────────────────┐ │ Jamba Serving Engine (vLLM) │ │ • Fused Mamba-SSM CUDA Kernels│ │ • PagedAttention for 1/8 KV │ │ • Expert Parallelism (EP=2) │ └──────────────┬────────────────┘ │ ▼ 3x Faster Autoregressive Decode

1. High-Performance Serving with vLLM

Modern releases of vLLM natively support Jamba hybrid models:

  • Ensure the fused causal-conv1d and mamba-ssm CUDA extensions are installed on host containers.
  • Start the server with FlashAttention-2 and continuous batching:
vllm serve ai21labs/AI21-Jamba-1.5-Mini \ --max-model-len 131072 \ --gpu-memory-utilization 0.90 \ --tensor-parallel-size 2 \ --port 8000

2. Sizing Expert Parallelism (EP) and Tensor Parallelism (TP)

Because Jamba has 52B total parameters, running unquantized FP16 requires approximately 105 GB of memory for model weights alone:

  • Optimal Cloud Node: A dual-GPU instance (e.g., AWS g5.12xlarge with 4x A10G 24GB, or p4d.24xlarge with 8x A100 80GB).
  • Set Tensor Parallelism $TP = 2$ to split weights across two 80GB GPUs. The remaining 55 GB of VRAM per GPU supports massive concurrent batch sizes even at 128k context lengths.

3. INT8 / INT4 Quantization

For single-GPU workstations or edge deployments:

  • Use AWQ or FP8 quantization on the MoE MLP layers.
  • Quantizing the 16 experts to 4-bit shrinks model weights from 105 GB down to 28 GB, allowing full Jamba-52B inference on a single NVIDIA RTX 4090 (24GB) or A5000 (32GB) with complete 128k context handling!

Review our flexible pricing tiers for compute options designed for long-context hybrid foundation models.

For comprehensive architectural specifications, integration blueprints, and deployment checklists, review our comprehensive developer documentation.

Frequently Asked Questions (FAQ)

What is the exact ratio of Mamba to Attention layers in Jamba?

Jamba uses a 1:7 ratio: for every 8 transformer layers in the model, 7 layers are Mamba selective state-space blocks, and exactly 1 layer is a standard multi-head self-attention block. This structure provides the $O(N)$ linear efficiency of Mamba while using periodic attention layers to anchor long-range associative recall.

How does Jamba manage to fit 140k tokens on a single 80GB GPU?

In standard transformers, every layer retains Key-Value cache vectors, causing memory to explode linearly with sequence length. In Jamba, because 7 out of 8 layers are Mamba blocks (which require zero KV cache and maintain only a constant $O(1)$ state vector), the total KV cache memory footprint is reduced by 87.5% (8x smaller). This frees up over 100 GB of GPU memory, allowing 140k tokens to fit in RAM on a single 80GB card.

How does Jamba compare to Mixtral 8x7B?

While both models utilize sparse Mixture-of-Experts (MoE) FFNs to maintain low active parameter counts during inference, Mixtral 8x7B uses 100% self-attention layers. Consequently, Mixtral inherits the full quadratic memory and latency penalty of transformers on long sequences. Jamba combines MoE with 1:7 Mamba interleaving, achieving 3x higher throughput on long sequences and supporting 256k tokens compared to Mixtral’s native 32k window.

Does Jamba require special CUDA kernels to achieve its speedups?

Yes. To achieve maximum throughput, Jamba relies on fused CUDA kernels for the Mamba selective state-space scan (mamba-ssm and causal-conv1d). These kernels keep intermediate state updates resident in fast on-chip GPU SRAM, avoiding slow round-trips to High-Bandwidth Memory (HBM).