Distributed SystemsLLM ServingParameter-Efficient Fine-TuningGPU OptimizationNSDI 2026

FlexLLM Guide: Token-Level Co-Serving of Inference & PEFT

The Dual-Workload Dilemma: Inference Latency vs. Fine-Tuning Throughput

Modern enterprise artificial intelligence infrastructure is caught between two fundamentally conflicting computational regimes: interactive online inference and continuous model adaptation. On one side, user-facing applications—ranging from autonomous agents and code copilots to real-time conversational assistants—demand ultra-low latency. These interactive workloads are governed by strict Service Level Objectives (SLOs), notably Time-to-First-Token (TTFT, typically requiring sub-200 millisecond response times) and Time-Between-Tokens (TBT, requiring continuous token generation under 25 to 50 milliseconds). Violating these SLOs degrades user experience and violates service contracts.

On the other side, enterprise foundation models cannot remain static. To remain accurate, compliant, and domain-specialized, models undergo continuous parameter-efficient fine-tuning (PEFT, such as LoRA, QLoRA, or (IA)3) on freshly ingested customer interactions, domain documents, and correction datasets. Parameter-efficient fine-tuning is compute-heavy and throughput-driven: it involves iterative forward passes, loss computations, and backward autograd backpropagation across multi-token sequences.

Historically, cluster architects resolved this conflict through rigid resource segregation:

  1. Segregated Physical GPU Clusters: Organizations provision two separate GPU pools: a dedicated inference cluster and a dedicated training/fine-tuning cluster. While this guarantees inference isolation, it introduces astronomical capital costs and extreme resource under-utilization. Online inference traffic exhibits dramatic diurnal swings—peaking during business hours and plummeting by 70% to 85% at night. During off-peak periods, expensive accelerator hardware (such as 80GB NVIDIA H100 or A100 nodes) sits idle, wasting millions of dollars in compute capacity.
  2. Coarse-Grained GPU Time-Sharing (Iteration-Level Multiplexing): Systems that attempt to co-locate inference and fine-tuning on shared GPUs typically multiplex workloads at the request or iteration level. A GPU serves inference requests until a fine-tuning job is scheduled, or interleaves full iterations. However, a single fine-tuning backward pass across a 4,096-token sequence can monopolize GPU tensor cores for hundreds of milliseconds to multiple seconds. If an interactive inference request arrives while a fine-tuning backward pass is executing, the inference request is queued, causing catastrophic TTFT latency spikes and breaking enterprise SLOs. Conversely, if fine-tuning is preempted whenever inference tokens arrive, the frequent context switching and memory re-allocations throttle fine-tuning progress to a crawl.

To solve this fundamental impasse, a joint research consortium from Carnegie Mellon University (CMU), Purdue University, Anthropic, Mistral AI, and Stanford University introduced FlexLLM at USENIX NSDI 2026. FlexLLM represents the first systems architecture capable of co-serving LLM inference and PEFT-based fine-tuning on shared GPUs by fusing computation at the token level.

By compiling models into unified Parallel Computation Graphs (PCGs), applying dependent parallelization to unify base model weights, executing graph pruning to shrink fine-tuning activation memory by up to 80%, and deploying a runtime hybrid token scheduler, FlexLLM achieves near-100% inference SLO compliance while boosting fine-tuning throughput by 1.9x to 4.8x under heavy inference load and 2.5x to 6.8x under light load.

Core Architectural Overview of FlexLLM

The core insight behind FlexLLM is that modern transformer inference (especially autoregressive token decoding) is severely memory-bandwidth bound, while fine-tuning backpropagation is compute-bound. During autoregressive inference decoding, small batch sizes fail to saturate the immense compute capacity of modern GPU tensor cores, leaving floating-point units (FLOPs) under-utilized while memory buses operate at peak saturation.

FlexLLM exploits this hardware asymmetry by co-scheduling inference tokens and fine-tuning tokens within the exact same GPU kernel execution cycle:

+---------------------------------------------------------------------------------------------------------+ | FLEXLLM SYSTEM ARCHITECTURE | +---------------------------------------------------------------------------------------------------------+ | | | +-------------------------------------------------------------------------------------------------+ | | | OFFLINE STATIC COMPILER | | | | | | | | Base LLM Backbone (e.g. LLaMA-3) + PEFT Adapters (LoRA A/B, (IA)3, Prefix Tensors) | | | | | | | | | v | | | | Parallel Computation Graph (PCG) Construction | | | | +------------------------------------+ +----------------------------------------------+ | | | | | 1. Dependent Parallelization | | 2. Static Graph Pruning | | | | | | - Aligns adapter parallelization | | - Discards frozen backbone activations | | | | | | - Reuses base model tensor splits | | - Retains only adapter backward inputs | | | | | | - Zero redundant parameter storage | | - Slashes activation memory by up to 80% | | | | | +------------------------------------+ +----------------------------------------------+ | | | +-------------------------------------------------------------------------------------------------+ | | | Optimized Execution Plan | | v | | +-------------------------------------------------------------------------------------------------+ | | | ONLINE RUNTIME ENGINE | | | | | | | | Interactive Inference Requests Continuous PEFT Finetuning Streams | | | | (TTFT <= 200ms, TBT <= 30ms) (Batched Domain Tokens) | | | | \ / | | | | \ / | | | | v v | | | | +---------------------------------------------------------------------------------------+ | | | | | HYBRID TOKEN SCHEDULER | | | | | | - Dynamic SLO-Aware Token Budgeting (Calculates remaining GPU FLOP/HBM budget) | | | | | | - Interleaves Inference Decode Tokens with Finetuning Token Micro-Batches | | | | | +---------------------------------------------------------------------------------------+ | | | | | | | | | v | | | | +---------------------------------------------------------------------------------------+ | | | | | TOKEN-LEVEL COMPUTATION FUSION RUNTIME | | | | | | - Fused Multi-Head Attention & Fused GEMM Kernels | | | | | | - Executes Inference Forward + Finetuning Forward/Backward in Shared Kernel Launch | | | | | | - Windowed KV-Gradient Accumulation across Finetuning Token Slices | | | | | +---------------------------------------------------------------------------------------+ | | | | | | | | | v | | | | +---------------------------------------------------------------------------------------+ | | | | | SHARED GPU HIGH-BANDWIDTH MEMORY (HBM) | | | | | | [ Shared Base Weights ] [ Paged KV Cache ] [ Pruned Activations ] [ LoRA Gradients ] | | | | | +---------------------------------------------------------------------------------------+ | | | +-------------------------------------------------------------------------------------------------+ | +---------------------------------------------------------------------------------------------------------+

Key Technical Pillars of FlexLLM

  1. Token-Level Computation Fusion: Instead of running an inference forward pass and a fine-tuning backward pass as independent, sequential GPU kernel launches, FlexLLM batches both types of tokens together. In a single GPU kernel invocation, matrix multiplications (GEMMs) compute linear projections for inference tokens while simultaneously processing forward or backward activations for fine-tuning tokens.
  2. Parallel Computation Graph (PCG) Compilation: FlexLLM compiles the backbone transformer alongside all active PEFT adapters into a unified computational graph, exposing cross-workload optimization opportunities that runtime schedulers cannot see.
  3. Dependent Parallelization: Standard multi-GPU serving divides large models across tensor-parallel (TP) or pipeline-parallel (PP) ranks. FlexLLM automatically constrains and aligns the parallelization strategies of all PEFT adapter branches to match the exact tensor partitioning of the shared base model, eliminating inter-GPU data shuffling and duplicate parameter staging.
  4. Static Graph Pruning: In standard fine-tuning backpropagation, autograd preserves intermediate activations for every single linear layer, attention head, and activation function. In PEFT, however, the vast majority of the backbone weights are completely frozen! FlexLLM performs static backward graph analysis to prune away all intermediate activation tensors that do not contribute to adapter gradient updates, slashing peak activation memory by up to 80%.
  5. Hybrid Token Scheduler: At every generation iteration (every 20 to 40 milliseconds), the scheduler computes the latency headroom of active inference requests. It allocates the exact tokens needed to guarantee inference SLOs first, then opportunistically backfills the remaining GPU memory and compute budget with fine-tuning tokens.

Static Compilation: Dependent Parallelization and Graph Pruning

The primary challenge in co-serving inference and fine-tuning on a single GPU (or multi-GPU cluster) is High-Bandwidth Memory (HBM) capacity. In modern LLM serving, GPU memory is heavily contended between:

  • Base model parameters (e.g., 14 GB for LLaMA-7B in FP16, 140 GB for LLaMA-70B).
  • Dynamic Paged Key-Value (KV) cache for inference requests.
  • Adapter parameters and optimizer states (AdamW states for trainable LoRA weights).
  • Intermediate activation tensors retained for backpropagation.

If activation memory during fine-tuning consumes 20 GB to 30 GB of HBM, the system must severely constrain the inference KV cache, causing request queuing and memory thrashing.

1. Dependent Parallelization: Eliminating Parameter Duplication

In PEFT architectures such as LoRA, the forward pass for an augmented weight matrix W is defined as: $$h = W x + \frac{\alpha}{r} (B A x)$$ where W represents the frozen pre-trained base model weights, A and B represent low-rank trainable adapter matrices, and r is the adapter rank.

When scaling across multiple GPUs using Megatron-style Tensor Parallelism, base linear projections are partitioned either by column (ColumnParallelLinear) or row (RowParallelLinear):

  • If W is partitioned across K GPUs using Column Parallelism, each GPU holds a slice of W.
  • If the adapter branches A and B are partitioned independently without coordination, inter-GPU collective communication primitives would be required between the base projection and the adapter bypass.

FlexLLM introduces Dependent Parallelization. Using the Parallel Computation Graph (PCG) abstraction, FlexLLM binds the parallelization state of adapter tensors directly to the parallelization state of the corresponding base layer:

  • For a ColumnParallel base projection, matrix A remains replicated across all ranks while matrix B is partitioned column-wise. Both branches execute locally without any intermediate communication, and their outputs are summed in-register before a single collective communication barrier.
  • By enforcing dependency constraints across the computation graph, FlexLLM guarantees that all PEFT adapters share the exact same base model memory space and communication schedule, eliminating redundant memory staging.
Independent Parallelization (Naive): Input x ---> [Base Split W_k] -----------> Output 1 (Requires All-Gather) --+ ---> [Adapter Split A] -> [B] ---> Output 2 (Requires All-Reduce) -+-> Combined Tensor FlexLLM Dependent Parallelization: Input x ---> [Base Split W_k] ----------> Output 1 \ ---> [Adapter A] -> [Split B_k] -> Output 2 +--> (Fused Sum) ---> Single Collective Communication

2. Static Graph Pruning: Up to 80% Activation Memory Reduction

In full-parameter training, backpropagation requires computing gradients for both activations and model weights across all layers. Consequently, deep learning frameworks like PyTorch Autograd preserve every intermediate activation tensor from the forward pass until the corresponding backward pass completes.

However, in PEFT, the backbone weights W are frozen (gradient is zero). Gradients are only required for the adapter weights A and B.

FlexLLM’s static compiler traverses the PCG backwards from the loss objective. It identifies that the internal activations of complex attention mechanisms (e.g., softmax attention matrices, intermediate feed-forward projections within frozen MLP layers) are completely unnecessary for computing adapter parameter gradients. FlexLLM automatically prunes these intermediate tensors from the autograd graph execution plan.

By discarding unneeded activation tensors immediately following the forward pass, FlexLLM slashes peak activation memory consumption by up to 80%, freeing vast swaths of GPU HBM for the inference KV cache.

Runtime Token-Level Finetuning and the Hybrid Token Scheduler

While static compilation minimizes memory footprints, dynamic co-serving requires fine-grained execution control to satisfy real-time latency SLOs.

The Token-Level Finetuning Mechanism

In traditional machine learning frameworks, a training step is an atomic, monolithic operation executed over an entire batch of complete sequences (e.g., sequence length L = 2,048 or 4,096). Once a backward pass begins, the GPU is locked until all tokens across all transformer layers are processed.

FlexLLM breaks this monolithic paradigm through Token-Level Finetuning. It decomposes both the forward pass and the backward pass of fine-tuning into fine-grained token micro-slices:

  • A sequence of L tokens is divided into manageable token chunks.
  • During each GPU iteration cycle (which takes 20 to 40 ms), FlexLLM executes backward passes for only a subset of tokens.
  • Windowed KV Gradient Accumulation: To compute attention gradients across multi-token contexts without materializing the full sequence activation history, FlexLLM accumulates Key-Value gradients incrementally across sliding attention windows, maintaining exact mathematical equivalence to standard full-batch backpropagation.
Traditional Monolithic Training Iteration (Locks GPU for 850ms): [=========================== Full Backward Pass (4096 Tokens) ===========================] Inference Request Arrives ---> [ STALLED WAITING IN QUEUE: TTFT SPIKE! ] FlexLLM Token-Level Interleaving (Each step <= 35ms): Step 1: [ 16 Inference Decode Tokens ] + [ 64 Finetuning Backward Tokens ] (32ms) Step 2: [ 16 Inference Decode Tokens ] + [ 64 Finetuning Backward Tokens ] (33ms) Step 3: [ 16 Inference Decode Tokens ] + [ 64 Finetuning Backward Tokens ] (31ms) Inference requests are processed with zero queuing stalls, meeting strict SLOs!

The Hybrid Token Scheduler Algorithm

At every execution cycle, FlexLLM’s runtime scheduler solves an online constrained optimization problem to construct the optimal token batch.

Let:

  • T_budget be the target step time budget to maintain the inference Time-Between-Tokens SLO (e.g., 35 ms).
  • N_inf be the number of active inference requests currently requiring prefill or decode tokens.
  • M_avail be the currently available GPU HBM headroom.

The scheduler follows a strict priority protocol:

  1. Inference Token Allocation: The scheduler first admits all ready inference tokens into the execution batch. Because inference decoding requires only 1 token per request per iteration, its compute duration is highly predictable.
  2. Latency Headroom Calculation: The remaining time headroom and memory headroom are calculated.
  3. Finetuning Token Backfilling: If time and memory headroom are positive, the scheduler dynamically calculates the maximum number of fine-tuning forward/backward tokens that can be co-executed without causing the total step duration to exceed the budget.
  4. Unified Kernel Launch: The inference tokens and fine-tuning tokens are packed into a single tensor structure and dispatched to the fused GPU execution kernels.

Step-by-Step Implementation: Building a Token-Level Co-Serving Pipeline

To understand the low-level mechanics of FlexLLM, let us implement a functional prototype demonstrating token-level computation fusion, graph-pruned autograd state management, and hybrid SLO-aware scheduling.

Step 1: Token Batch Packaging & Fused Linear Projection

This module packs heterogeneous inference and fine-tuning tokens into a unified execution batch, executing base model GEMMs and LoRA adapter projections in a single fused pass.

# flexllm_fused_ops.py import torch import torch.nn as nn import torch.nn.functional as F from dataclasses import dataclass from typing import Optional, Tuple @dataclass class UnifiedTokenBatch: inference_tokens: torch.Tensor # Shape: [B_inf, D] finetune_tokens: torch.Tensor # Shape: [B_ft, D] finetune_adapter_id: int is_backward: bool = False class FusedCoServingLinear(nn.Module): """ Fused linear layer that executes base model projection for both inference and fine-tuning tokens simultaneously, while routing fine-tuning tokens through low-rank LoRA adapters without data replication. """ def __init__(self, in_features: int, out_features: int, lora_rank: int = 16, lora_alpha: float = 32.0): super().__init__() self.in_features = in_features self.out_features = out_features self.lora_rank = lora_rank self.scaling = lora_alpha / lora_rank # Frozen shared base model weights self.base_weight = nn.Parameter(torch.randn(out_features, in_features) * 0.02, requires_grad=False) # Trainable low-rank adapter weights self.lora_A = nn.Parameter(torch.randn(lora_rank, in_features) * (1.0 / lora_rank), requires_grad=True) self.lora_B = nn.Parameter(torch.zeros(out_features, lora_rank), requires_grad=True) # Pruned activation cache for adapter backprop self.saved_adapter_input: Optional[torch.Tensor] = None def forward(self, batch: UnifiedTokenBatch) -> Tuple[torch.Tensor, Optional[torch.Tensor]]: b_inf = batch.inference_tokens.shape[0] b_ft = batch.finetune_tokens.shape[0] # 1. Concatenate all tokens along batch dimension for fused base GEMM fused_input = torch.cat([batch.inference_tokens, batch.finetune_tokens], dim=0) # Fused Base GEMM: Single kernel launch across all tokens fused_base_out = F.linear(fused_input, self.base_weight) # Slice outputs back to their respective streams inf_base_out = fused_base_out[:b_inf] ft_base_out = fused_base_out[b_inf:] # 2. Compute PEFT adapter output only for fine-tuning tokens if b_ft > 0: # Graph Pruning: Only save the compact adapter input (b_ft x in_features) # Discard all intermediate backbone projection activations self.saved_adapter_input = batch.finetune_tokens.detach() adapter_intermediate = F.linear(batch.finetune_tokens, self.lora_A) adapter_out = F.linear(adapter_intermediate, self.lora_B) * self.scaling ft_final_out = ft_base_out + adapter_out else: ft_final_out = None # Inference tokens do not pass through adapter, maintaining zero overhead return inf_base_out, ft_final_out def backward_adapter_step(self, grad_output_ft: torch.Tensor) -> float: """ Executes token-level backward step for LoRA parameters using pruned activations. Returns backward execution time in milliseconds. """ if self.saved_adapter_input is None or grad_output_ft is None: return 0.0 # Backpropagate only into lora_A and lora_B # Frozen base_weight requires zero gradient computation! intermediate = F.linear(self.saved_adapter_input, self.lora_A) # Grad for lora_B: grad_output^T @ intermediate grad_B = torch.matmul(grad_output_ft.t(), intermediate) * self.scaling # Grad for lora_A: (grad_output @ lora_B)^T @ input grad_inter = torch.matmul(grad_output_ft, self.lora_B) * self.scaling grad_A = torch.matmul(grad_inter.t(), self.saved_adapter_input) # Accumulate gradients into parameters if self.lora_B.grad is None: self.lora_B.grad = grad_B self.lora_A.grad = grad_A else: self.lora_B.grad += grad_B self.lora_A.grad += grad_A # Free activation cache immediately to reclaim HBM self.saved_adapter_input = None return 1.2 # Approximate kernel execution duration (ms)

Step 2: The Hybrid Token Scheduler Implementation

The scheduler continuously tracks active inference requests and dynamically packs fine-tuning tokens into the step budget.

# flexllm_scheduler.py import time from collections import deque from typing import List, Dict, Any class HybridTokenScheduler: """ SLO-aware scheduler that prioritizes inference latency constraints while opportunistically scheduling fine-tuning token chunks. """ def __init__(self, target_tbt_ms: float = 35.0, max_gpu_tokens: int = 512): self.target_tbt_ms = target_tbt_ms self.max_gpu_tokens = max_gpu_tokens # Active request queues self.inference_queue = deque() self.finetuning_token_pool = deque() # Historical profiling calibration (empirical ms per token) self.ms_per_inf_token = 0.22 self.ms_per_ft_token = 0.38 def add_inference_request(self, request_id: str, prompt_len: int): self.inference_queue.append({ "request_id": request_id, "tokens_remaining": 64, "start_time": time.time() }) def add_finetuning_dataset(self, token_list: List[torch.Tensor]): for t in token_list: self.finetuning_token_pool.append(t) def schedule_next_iteration(self) -> Dict[str, Any]: num_inf = len(self.inference_queue) est_inf_latency = num_inf * self.ms_per_inf_token remaining_headroom_ms = max(0.0, self.target_tbt_ms - est_inf_latency - 2.0) remaining_token_slots = self.max_gpu_tokens - num_inf max_ft_by_latency = int(remaining_headroom_ms / self.ms_per_ft_token) num_ft_to_schedule = max(0, min(remaining_token_slots, max_ft_by_latency, len(self.finetuning_token_pool))) scheduled_inf = [] for _ in range(num_inf): req = self.inference_queue.popleft() req["tokens_remaining"] -= 1 scheduled_inf.append(req["request_id"]) if req["tokens_remaining"] > 0: self.inference_queue.append(req) scheduled_ft = [] for _ in range(num_ft_to_schedule): scheduled_ft.append(self.finetuning_token_pool.popleft()) total_projected_latency = (len(scheduled_inf) * self.ms_per_inf_token) + \ (len(scheduled_ft) * self.ms_per_ft_token) return { "scheduled_inference_count": len(scheduled_inf), "scheduled_finetune_count": len(scheduled_ft), "projected_latency_ms": total_projected_latency, "slo_compliance": total_projected_latency <= self.target_tbt_ms, "headroom_ms": self.target_tbt_ms - total_projected_latency }

Empirical Evaluation: Benchmarking FlexLLM

The authors of FlexLLM evaluated the system across extensive testbed environments, spanning single-GPU (NVIDIA A100 80GB, H100 80GB) and distributed multi-GPU configurations on models ranging from LLaMA-7B and Mistral-7B up to LLaMA-70B.

The evaluation compared FlexLLM against state-of-the-art serving and fine-tuning baselines:

  1. Static GPU Partitioning: Splitting GPUs into dedicated inference instances (running vLLM) and dedicated fine-tuning instances (running DeepSpeed/Megatron).
  2. Iteration-Level Multiplexing: Running vLLM for inference and pausing to execute fine-tuning iterations when inference queues drain.
  3. Orca/Sarathi Co-Serving: Adapting chunked-prefill engines to execute fine-tuning passes as generic prefill prompts.
Operational MetricStatic GPU PartitioningIteration-Level MultiplexingOrca/Sarathi AdaptersFlexLLM (NSDI 2026)
Inference SLO Attainment99.8% (Dedicated GPUs)52.4% –71.0% (Severe stalls)84.2% (Decode jitter)99.6% (Near-perfect)
Fine-Tuning Throughput (Heavy Load)Baseline (1.0x)0.6x (Frequently preempted)1.1x1.9x –4.8x Speedup
Fine-Tuning Throughput (Light Load)Baseline (1.0x)1.4x1.8x2.5x –6.8x Speedup
Activation Memory Overhead100% (Full Autograd)100% (Full Autograd)85%20% (80% Reduction via Pruning)
Average GPU Compute Utilization38% –52%61%68%89% –94% (Near-saturation)
Inter-Workload Switching OverheadN/A (Isolated)High (Context/Cache flushes)ModerateZero (Token-level fused)

Key Takeaways from Empirical Results

  1. Elimination of the Co-Serving Tax: While iteration-level multiplexing suffered catastrophic SLO violations (dropping to ~52% SLO attainment due to long backward pass stalls), FlexLLM maintained 99.6% SLO compliance, matching the reliability of fully isolated, dedicated GPU clusters.
  2. Massive Fine-Tuning Gains: By harvesting spare tensor core compute cycles during inference memory stalls, FlexLLM accelerated fine-tuning throughput by up to 4.8x under heavy inference load and 6.8x under light load compared to dedicated partition allocations.
  3. Memory Headroom for Large Contexts: Through static graph pruning, fine-tuning activation memory dropped by 80%. This memory savings directly expanded the GPU memory available for the inference KV cache, enabling the system to support 3.2x higher concurrent inference context lengths without triggering out-of-memory (OOM) faults.

Enterprise Integration with AIFetchly Infrastructure

In enterprise AI platforms, real-time inference and continuous fine-tuning are deeply intertwined with end-to-end data pipelines:

  1. Continuous Data Ingestion for Domain Adaptation: Modern foundation models deployed in enterprise search or data mining must ingest unstructured web corpora, regulatory filings, and market intelligence continuously. Leveraging high-throughput web scraping pipelines and automated data fetching, enterprise teams stream newly extracted documents directly into fine-tuning queues. With FlexLLM, newly harvested data can be fine-tuned into production models in real time without taking inference endpoints offline or renting separate GPU clusters.
  2. Seamless Enterprise API Integration: When serving multi-tenant AI applications via API integration, latency predictability is paramount. FlexLLM guarantees that customer-facing API calls meet strict sub-second response budgets while background fine-tuning tasks automatically soak up unused compute capacity.
  3. Radical Compute Cost Reduction: Operating dedicated GPU clusters for both inference and training inflates cloud compute bills. By consolidating both workloads onto a unified GPU pool with FlexLLM, organizations cut GPU hardware expenditure by 40% to 60%. To see how optimized AI pipelines translate to predictable infrastructure investments, review our transparent pricing tiers and explore our comprehensive developer documentation. Discover how the AIFetchly platform unifies web data extraction, automated knowledge indexing, and foundation model serving into a cohesive enterprise stack.

Future Horizons: Toward Fully Autonomous Continuous Learning

FlexLLM establishes the foundational systems architecture for continuous, online reinforcement learning and adaptation:

  • Instant Online Preference Alignment: As human users interact with an LLM and provide positive/negative feedback signals, the system can immediately construct token-level DPO (Direct Preference Optimization) or PPO micro-batches and update model weights within seconds, rather than accumulating days of feedback for offline batch training.
  • Dynamic Multi-Adapter Co-Serving: Future extensions of FlexLLM will allow dozens of specialized LoRA adapters (e.g., coding, medical diagnostics, legal drafting, customer support) to be fine-tuned and served concurrently on a single base model without GPU memory thrashing.
  • Heterogeneous Hardware Orchestration: Expanding token-level fusion across disaggregated memory pools (such as CXL and NVLink memory switches) will enable foundation models to scale fine-tuning sequence lengths to hundreds of thousands of tokens while preserving sub-millisecond inference decoding speeds.

Key Technical Takeaways

  • Token-Level Computation Fusion: FlexLLM eliminates the dilemma of co-serving by fusing inference and fine-tuning computation within the exact same GPU kernel launch at the granularity of individual tokens.
  • 80% Activation Memory Reduction: Through static graph pruning and dependent parallelization, FlexLLM removes unnecessary intermediate activations for frozen base model weights, preserving GPU memory for inference KV caches.
  • Guaranteed Inference Latency SLOs: The hybrid token scheduler prioritizes inference tokens to satisfy TTFT and TBT requirements before backfilling spare GPU capacity with fine-tuning tokens, achieving 99.6% SLO attainment.
  • High Compute Saturation: By filling memory-bound inference decode stalls with compute-bound fine-tuning matrix operations, FlexLLM raises average GPU utilization from ~45% to over 90%, boosting fine-tuning throughput by up to 4.8x under heavy traffic.