deepserveserverless-aillm-servingflowservenpu-forkdistributed-systemsusenix-atc-2025tutorial

DeepServe Guide: Serverless LLM Serving at Hyperscale

DeepServe Architecture: Request-Job-Task Model, FLOWSERVE Microkernel Engine, Sub-Second NPU-Fork Scaling, and Hybrid PD Scheduling

The Serverless Dilemma in Modern LLM Infrastructure

In modern cloud computing and enterprise artificial intelligence, Large Language Model (LLM) workloads are characterized by extreme temporal volatility. Production platforms—such as autonomous code copilots, multi-tenant document extraction pipelines, and real-time knowledge synthesis engines built by aiFetchly—experience dramatic swings in request volume throughout the day. Traffic surges during business hours can exceed overnight troughs by more than an order of magnitude.

Historically, cloud infrastructure architects turned to serverless computing (e.g., AWS Lambda, Google Cloud Run) to navigate traffic burstiness. By automatically allocating compute resources in response to incoming queries and scaling down to zero during idle periods, serverless architectures eliminate idle hardware costs while providing elastic scalability.

However, when applied to modern generative AI, traditional serverless paradigms collide with severe physical and architectural barriers:

  1. The Crushing Cold-Start Penalty: Conventional serverless functions spin up in tens of milliseconds. In stark contrast, spinning up an LLM inference replica (such as Llama-3-70B or DeepSeek-V3) requires allocating dedicated GPU/NPU memory, initializing heavyweight runtime environments (CUDA, PyTorch, NCCL), and streaming tens or hundreds of gigabytes of model weights from remote storage or host DRAM over PCIe. On standard Kubernetes clusters, a cold start frequently demands 90 to 300 seconds—rendering on-demand scaling completely useless for interactive user queries.
  2. Monolithic Serving Engine Inflexibility: Popular open-source serving runtimes (such as vanilla vLLM, TGI, and TensorRT-LLM) are designed around static, long-running daemon architectures. Their control planes and execution engines are tightly coupled, making it difficult to pause, preempt, dynamically scale, or repurpose accelerators across heterogeneous workloads.
  3. Rigid Parallelism and Hardware Fragmentation: Enterprise workloads do not consist solely of standard text generation. Production clusters must interleave interactive inference, agentic multi-turn loops, offline batch document processing, and lightweight fine-tuning (e.g., LoRA post-training). Statically partitioning hardware into dedicated inference pools and fine-tuning pools leads to severe resource fragmentation and stranded accelerator capacity.
The Serverless LLM Serving Dilemma: +-----------------------------------------------------------------------------------+ | Approach 1: Static Over-Provisioning (Dedicated Cluster Instances) | | • Provision GPU/NPU nodes for peak traffic (e.g., 64x H100 or Ascend 910B). | | • Instant response during surges; zero cold-start latency. | | ❌ Catastrophic Financial Waste: | | - Average cluster utilization languishes at 15% - 25% during off-peak hours. | | - Millions of dollars wasted on idle High-Bandwidth Memory (HBM) and power. | +-----------------------------------------------------------------------------------+ | Approach 2: Traditional Cloud Auto-Scaling (HPA on Vanilla Kubernetes) | | • Automatically add Pod replicas when incoming request queues grow. | | ❌ Catastrophic Cold-Start Latency Collapse: | | - Container image pull + PyTorch init + weight loading takes 2 to 5 minutes! | | - Traffic spikes vanish or time out long before new replicas become healthy. | | - Users experience severe SLO violations (>60% query dropouts). | +-----------------------------------------------------------------------------------+

To dismantle this scalability bottleneck, systems researchers from Peking University and Huawei Cloud introduced DeepServe (presented at USENIX ATC 2025 in DEEPSERVE: Serverless Large Language Model Serving at Scale, arXiv:2501.14417).

DeepServe is a unified serverless AI serving platform designed from the ground up for hyperscale cloud environments. By introducing a novel Request-Job-Task abstraction, a modular microkernel serving engine named FLOWSERVE, a hybrid scheduling policy that balances prefill-decode disaggregation with co-location, and sub-second NPU-Fork scaling, DeepServe achieves what was previously thought impossible: scaling up to 64 accelerator instances in parallel within seconds.

In this technical guide, we dissect the architecture of DeepServe, examine its core mechanisms, explore its benchmark results across massive production traces, and build a working end-to-end Python implementation.

DeepServe Architectural Foundations: The Request-Job-Task Model

At the core of DeepServe is a new serverless programming and operational abstraction: the Request-Job-Task model.

Conventional serving engines treat incoming HTTP requests as monolithic end-to-end execution units. This tight coupling prevents the scheduler from decomposing long queries, isolating prefill from decoding, or interleaving training workloads with inference. DeepServe decouples execution into three hierarchical layers:

The Request-Job-Task Hierarchical Abstraction: ┌────────────────────────────────────────────────────────────────────────┐ | User Request Layer (Client API) | | • OpenAI-compatible endpoints, Fine-tuning jobs, Agent workflows | └──────────────────────────────────┬─────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────────────────┐ | Serverless Job Layer (Orchestration) | | • Translates incoming requests into typed Directed Acyclic Graphs | | • Types: PrefillJob, DecodeJob, PostTrainJob, AgentEvaluationJob | └──────────────────────────────────┬─────────────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────────────────────┐ | Execution Task Layer (Accelerator Workers) | | • Fine-grained, atomic computation kernels scheduled on NPUs/GPUs | | • Task 1: Prompt Prefill GEMM Task 2: KV Cache Block Migration | | • Task 3: Batched Token Decode Task 4: Gradient Backward Pass | └────────────────────────────────────────────────────────────────────────┘

1. Request Layer

The Request layer exposes standard industry APIs (OpenAI chat completions, embeddings, and LoRA tuning endpoints). It handles client authentication, rate limiting, and priority tag assignment (e.g., interactive tier vs. background tier).

2. Job Layer

When a request arrives, the Job orchestrator decomposes it into typed serverless jobs:

  • PrefillJob: Responsible for prompt evaluation and initial KV cache generation.
  • DecodeJob: Responsible for multi-step token generation.
  • PostTrainJob: Represents lightweight fine-tuning tasks (e.g., parameter-efficient tuning or reward model updates).
  • AgentToolJob: Represents multi-stage reasoning workflows where intermediate results trigger external tool calls.

By decoupling the prefill and decode stages into separate job entities, DeepServe can route them to specialized accelerator pools or dynamically reschedule them based on cluster load.

3. Task Layer

Jobs are compiled into discrete, atomic Execution Tasks. Each task is an SPMD (Single Program, Multiple Data) execution unit with explicit input dependencies, resource requirements (e.g., number of NPU cores, HBM memory footprint), and priority levels.

The Task layer allows DeepServe’s low-level scheduler to interleave tasks from different jobs onto the same accelerator hardware, maximizing compute efficiency and preventing head-of-line blocking.

FLOWSERVE Engine: Microkernel Architecture for Accelerator Runtimes

Traditional serving systems (such as vLLM or Hugging Face TGI) are structured as monolithic daemons: memory management, request scheduling, batch formation, and kernel execution are compiled into a single unified process. While effective for static deployments, this monolithic design makes it difficult to implement fine-grained preemption, fast scaling, or dynamic hardware reassignment.

To solve this, DeepServe introduces FLOWSERVE, an in-house serving engine built on a microkernel-inspired architecture.

FLOWSERVE Microkernel Architecture: +-----------------------------------------------------------------------------------+ | User Space / Control Plane | | ┌─────────────────────────┐ ┌─────────────────────────┐ ┌──────────────────┐ | | │ Dynamic Batching Policy │ │ KV Cache Paging Policy │ │ Scheduling Policy│ | | └─────────────────────────┘ └─────────────────────────┘ └──────────────────┘ | +-----------------------------------------------------------------------------------+ │ Fast IPC / Shared Memory Ring Buffer ▼ +-----------------------------------------------------------------------------------+ | FLOWSERVE Microkernel Core | | ┌─────────────────────────────────────────────────────────────────────────────┐ | | │ Base Primitives: │ | | │ • Atomic Accelerator Memory Allocator (HBM Block Virtualization) │ | | │ • SPMD Execution Dispatcher & Kernel Launch Queue │ | | │ • Low-Latency Inter-Device Communication Channel (HCCS / NVLink) │ | | │ • Task Preemption and Context Switch Engine │ | | └─────────────────────────────────────────────────────────────────────────────┘ | +-----------------------------------------------------------------------------------+ │ ▼ +-----------------------------------------------------------------------------------+ | Hardware Execution Plane (NPU / GPU) | | [Device 0: Ascend/H100] [Device 1: Ascend/H100] [Device 2: Ascend/H100] | +-----------------------------------------------------------------------------------+

1. Separation of Policy and Mechanism

In FLOWSERVE, high-level serving policies—such as continuous batching algorithms, KV cache eviction rules, and request priority routing—are executed as pluggable user-space services. The microkernel core provides only the minimal, highly optimized primitives necessary for accelerator interaction:

  • Zero-Copy Memory Allocation: Manages non-contiguous HBM pages with sub-microsecond allocation latency.
  • Hardware-Centric Dispatching: Directly submits compute graphs to hardware queues, bypassing Python runtime overhead.
  • Low-Latency Inter-Device IPC: Facilitates rapid tensor exchange across Huawei HCCS (High-Performance Computing Cluster System) or NVIDIA NVLink fabrics.

2. NPU-Centric Execution and SPMD Parallelism

Traditional distributed serving frameworks rely on host CPUs to coordinate every step of distributed inference, resulting in host-device synchronization bottlenecks. FLOWSERVE implements accelerator-centric execution: once a computation graph is launched, synchronization signals between tensor-parallel workers are handled directly by the accelerators through hardware barrier registers, keeping host CPU overhead near zero.

Solving the Cold-Start Crisis: NPU-Fork, Pre-Warming, and DRAM Staging

The defining technical breakthrough of DeepServe is its ability to scale from zero to 64 active instances in less than 2 seconds.

In standard cloud environments, initializing an 8-GPU/NPU inference worker takes minutes:

  1. Kubernetes Pod scheduling and container startup (10–25s).
  2. Python and PyTorch runtime initialization (15–30s).
  3. Driver and accelerator device context initialization (10–20s).
  4. Loading 140 GB of model weights from remote storage into host memory (30–90s).
  5. Copying weights from host memory over PCIe into GPU HBM (15–30s).

DeepServe eliminates every stage of this latency bottleneck through four coordinated mechanisms: Pre-Warmed Task Executors, DRAM Pre-Loading, High-Speed Inter-Accelerator Staging, and NPU-Fork.

The Anatomy of DeepServe Sub-Second Scaling: Traditional Kubernetes Cold Start: ~180 to 300 Seconds ┌──────────────┬──────────────────┬─────────────────┬────────────────────┬──────────┐ | Container OS | PyTorch Runtime | Driver / Device | Weight Read Remote | PCIe HBM | | Startup (20s)| Initialization | Context (15s) | Storage (60-120s) | Copy(25s)| └──────────────┴──────────────────┴─────────────────┴────────────────────┴──────────┘ DeepServe Sub-Second Scaling: < 2.0 Seconds (64 Instances!) ┌───────────────────────────────────────┬───────────────────────────────────────────┐ | Pre-Warmed Pods & Task Executors (TE) | NPU-Fork: Zero-Copy Accelerator Memory | | • Pre-initialized Python/PyTorch env | • Pre-loaded DRAM weights broadcast | | • Pre-allocated driver contexts | • Instant virtual memory clone via IPC | | • Latency: ~50 milliseconds | • Latency: < 1.5 seconds across 64 NPUs | └───────────────────────────────────────┴───────────────────────────────────────────┘

1. Pre-Warmed Pods and Task Executors (TEs)

DeepServe maintains a pool of generic, pre-warmed container pods known as Task Executors (TEs). These executors have their base OS, Python interpreters, CUDA/CANN libraries, and driver contexts pre-initialized at cluster boot time. When a scaling event occurs, the scheduler does not create a new pod; it simply binds an idle TE to the target model task in under 50 milliseconds.

Instead of downloading model checkpoints from remote object stores (e.g., S3 or Ceph) during scaling events, DeepServe caches popular model weights in host system DRAM across cluster nodes. When an instance spins up, weights are transferred from local DRAM or adjacent nodes via high-speed RDMA networks, bypassing disk I/O bottlenecks.

3. NPU-Fork: Instant Accelerator Process Cloning

In standard operating systems, the fork() system call creates a new process in milliseconds by sharing the parent process’s memory pages via Copy-on-Write (CoW). DeepServe introduces NPU-Fork, extending this concept to AI accelerator hardware.

When a model is first loaded on a node, a parent “Template Worker” holds the model weights in accelerator HBM and registers memory handles with the host kernel driver. When new worker instances are needed:

  1. DeepServe forks new child worker processes that inherit the parent’s virtual address mapping.
  2. The child workers map the parent’s read-only weight buffers directly into their own device contexts using inter-process memory sharing (IPC).
  3. Only the dynamic state (KV cache and scratchpad activation buffers) is allocated fresh for each child.

By sharing static model parameters across instances on the same node, NPU-Fork eliminates weight copying entirely. New worker instances become operational in under 1.5 seconds, enabling true sub-second serverless scaling.

Hybrid Scheduling: Balancing PD-Disaggregated and PD-Colocated Topologies

Recent serving breakthroughs (such as DistServe, Splitwise, and Mooncake) advocate for Prefill-Decode (PD) Disaggregation, splitting clusters into dedicated prefill nodes and dedicated decode nodes.

While pure PD disaggregation prevents heavy prefill computations from interrupting time-sensitive decode steps, DeepServe’s empirical analysis of production workloads reveals two major flaws in pure disaggregation:

  1. Static Pool Sizing Inefficiencies: Because the ratio of prompt tokens to output tokens fluctuates wildly across requests (e.g., long document summarization has high prompt/output ratios, whereas code generation has low prompt/output ratios), fixed prefill/decode instance ratios inevitably result in one pool sitting idle while the other is saturated.
  2. Decode-Length Uncertainty: When a request finishes prefill, the scheduler cannot predict how many tokens it will generate. Routing a request to a dedicated decode node based on inaccurate length estimates leads to memory fragmentation and load imbalance.

To resolve this, DeepServe implements a Hybrid Scheduling Policy that combines PD-disaggregated instances with PD-colocated instances.

DeepServe Hybrid Cluster Topology: +-----------------------------------------------------------------------------------+ | Incoming Request Stream | +-----------------------------------------------------------------------------------+ │ ▼ +-----------------------------------------------------------------------------------+ | DeepServe Multi-Tier Hybrid Scheduler | | • Workload Profiler: Evaluates prompt length & predicted decode horizon | | • Locality & Cache Matcher: Checks prefix KV cache availability | +-----------------------------------------------------------------------------------+ │ │ │ ▼ ▼ ▼ ┌─────────────────────────┐ ┌───────────────────────────┐ ┌─────────────────────┐ | Dedicated Prefill Pool | | Hybrid Colocated Pool | | Dedicated Decode | | • High TP (TP=4 or 8) | | • Dynamic Prefill/Decode | | Pool | | • Compute-dense GEMMs | | • Absorbs burst variance | | • Low TP (TP=1 or 2)| | • Slashes initial TTFT | | • Eliminates stranded VRAM| | • High batch size | └─────────────────────────┘ └───────────────────────────┘ └─────────────────────┘

1. Three-Tier Instance Architecture

DeepServe partitions its serving fleet into three cooperative tiers:

  • Dedicated Prefill Pool: Configured with high tensor parallelism (e.g., $TP=8$) to minimize TTFT for large prompts.
  • Dedicated Decode Pool: Configured with low tensor parallelism (e.g., $TP=1$ or $TP=2$) and large batch sizes to maximize memory bandwidth utilization.
  • Hybrid Colocated Pool: Dynamically executes both prefill and decode phases using continuous batching. This pool acts as an elastic buffer, absorbing overflow traffic from whichever dedicated pool is currently experiencing a burst.

2. Multi-Objective Scheduling Algorithm

When scheduling an incoming request $r$, the DeepServe scheduler evaluates a joint utility score across candidate instances $i$:

$$\text{Score}(r, i) = \alpha \cdot \text{Locality}(r, i) + \beta \cdot \text{Load}(i) + \gamma \cdot \text{SLO_Margin}(r, i)$$

where:

  • $\text{Locality}(r, i)$ measures prefix KV cache reuse (e.g., system prompt matching via Radix trees).
  • $\text{Load}(i)$ reflects current HBM memory pressure and pending queue depth.
  • $\text{SLO_Margin}(r, i)$ estimates the probability that instance $i$ will satisfy the request’s TTFT and TPOT deadlines based on historical execution profiling.

Complete End-to-End Implementation: Building a Serverless LLM Orchestrator

To demonstrate how DeepServe coordinates serverless job execution, NPU-Fork scaling, and hybrid scheduling, let us examine a complete, production-grade Python simulation.

This implementation models:

  1. RequestJobTask: The hierarchical serverless task DAG.
  2. NPUForkManager: Simulating instant accelerator memory cloning.
  3. HybridPDScheduler: Joint scheduling across disaggregated and colocated worker pools.
  4. DeepServeClusterSimulator: Simulating bursty traffic scaling from 4 to 64 instances in seconds.
""" DeepServe: Serverless Large Language Model Serving at Scale Reference Architecture and Simulator (USENIX ATC 2025) """ import time import math import heapq from dataclasses import dataclass, field from enum import Enum from typing import Dict, List, Optional, Tuple class JobType(Enum): PREFILL = "prefill" DECODE = "decode" POST_TRAIN = "post_train" class WorkerRole(Enum): PREFILL_DEDICATED = "prefill_dedicated" DECODE_DEDICATED = "decode_dedicated" HYBRID_COLOCATED = "hybrid_colocated" @dataclass class ServerlessTask: task_id: str job_id: str job_type: JobType prompt_tokens: int output_tokens: int arrival_time: float slo_deadline_ms: float completed_tokens: int = 0 actual_latency_ms: float = 0.0 is_completed: bool = False @dataclass class AcceleratorWorker: worker_id: str role: WorkerRole tp_level: int is_active: bool = True assigned_tasks: List[ServerlessTask] = field(default_factory=list) hbm_capacity_gb: float = 80.0 hbm_used_gb: float = 0.0 class NPUForkManager: """ Simulates DeepServe's NPU-Fork mechanism: Clones parent accelerator memory contexts without redundant weight loading. """ def __init__(self, model_weight_gb: float = 140.0): self.model_weight_gb = model_weight_gb self.prewarmed_te_pool: List[str] = [f"te_worker_{i}" for i in range(128)] def fork_instance(self, target_worker_id: str, role: WorkerRole, tp_level: int) -> Tuple[AcceleratorWorker, float]: """ Performs NPU-Fork: maps parent memory handles via IPC. Returns the spawned worker and the cold start latency in milliseconds. """ # Pre-warmed TE bind + NPU memory handle mapping takes under 1.5 seconds cold_start_latency_ms = 120.0 + (tp_level * 15.0) # ~150ms in DeepServe! new_worker = AcceleratorWorker( worker_id=target_worker_id, role=role, tp_level=tp_level, is_active=True, hbm_used_gb=self.model_weight_gb / tp_level ) return new_worker, cold_start_latency_ms class HybridPDScheduler: """ Coordinates request dispatching across Prefill-Dedicated, Decode-Dedicated, and Hybrid-Colocated worker pools. """ def __init__(self, initial_workers: List[AcceleratorWorker]): self.workers = initial_workers self.task_queue: List[ServerlessTask] = [] self.fork_mgr = NPUForkManager() self.completed_tasks: List[ServerlessTask] = [] def submit_request(self, task: ServerlessTask): self.task_queue.append(task) def auto_scale_workers(self, current_time: float): """ Evaluates queue pressure. If backlog threatens SLO deadlines, executes NPU-Fork to spawn new instances in parallel. """ pending_prefills = [t for t in self.task_queue if t.job_type == JobType.PREFILL] if len(pending_prefills) >= 8 and len(self.workers) < 64: scale_count = min(8, 64 - len(self.workers)) print(f"[{current_time:.2f}s] [DeepServe Autoscaler] Surge detected! Forking {scale_count} instances via NPU-Fork...") for k in range(scale_count): wid = f"forked_worker_{len(self.workers)}" worker, lat = self.fork_mgr.fork_instance(wid, WorkerRole.HYBRID_COLOCATED, tp_level=2) self.workers.append(worker) print(f"[{current_time:.2f}s] [DeepServe Autoscaler] Scale-out complete. Active workers: {len(self.workers)}.") def step(self, current_time: float, dt_ms: float = 20.0): # Check auto-scaling triggers self.auto_scale_workers(current_time) # Dispatch queued tasks to matching workers for worker in self.workers: while self.task_queue and len(worker.assigned_tasks) < 8: # Match task type to worker role candidate = None for t in self.task_queue: if worker.role == WorkerRole.PREFILL_DEDICATED and t.job_type == JobType.PREFILL: candidate = t break elif worker.role == WorkerRole.DECODE_DEDICATED and t.job_type == JobType.DECODE: candidate = t break elif worker.role == WorkerRole.HYBRID_COLOCATED: candidate = t break if candidate: self.task_queue.remove(candidate) worker.assigned_tasks.append(candidate) else: break # Execute step on assigned tasks if not worker.assigned_tasks: continue for t in list(worker.assigned_tasks): if t.job_type == JobType.PREFILL: # Prefill completed in one step t.job_type = JobType.DECODE t.completed_tokens = 1 else: t.completed_tokens += 1 if t.completed_tokens >= t.output_tokens: t.is_completed = True t.actual_latency_ms = (current_time - t.arrival_time) * 1000.0 worker.assigned_tasks.remove(t) self.completed_tasks.append(t) if __name__ == "__main__": print("=== Initializing DeepServe Serverless Cluster Simulation ===") initial_pool = [ AcceleratorWorker("prefill_0", WorkerRole.PREFILL_DEDICATED, tp_level=4), AcceleratorWorker("prefill_1", WorkerRole.PREFILL_DEDICATED, tp_level=4), AcceleratorWorker("decode_0", WorkerRole.DECODE_DEDICATED, tp_level=2), AcceleratorWorker("decode_1", WorkerRole.DECODE_DEDICATED, tp_level=2), ] scheduler = HybridPDScheduler(initial_pool) # Ingest synthetic burst of 32 requests sim_time = 0.0 for i in range(32): scheduler.submit_request(ServerlessTask( task_id=f"req_{i}", job_id=f"job_{i}", job_type=JobType.PREFILL, prompt_tokens=2048, output_tokens=64, arrival_time=sim_time, slo_deadline_ms=500.0 )) # Advance simulation while sim_time < 3.0 and (scheduler.task_queue or any(w.assigned_tasks for w in scheduler.workers)): scheduler.step(sim_time) sim_time += 0.05 print(f"\nSimulation finished at T={sim_time:.2f}s.") print(f"Total Completed Tasks: {len(scheduler.completed_tasks)}") print(f"Final Cluster Worker Count: {len(scheduler.workers)}")

To run this simulation, execute the script with Python 3:

python3 deepserve_simulation.py

Benchmark Evaluation: Scalability, Cold Start, and Cluster Utilization

The authors of DeepServe evaluated the platform extensively on Huawei Cloud production clusters equipped with Ascend 910B NPUs and NVIDIA H100 SXM5 systems, testing across production workloads that encompass LLM inference, multi-agent workflows, and LoRA fine-tuning.

Cold-Start Latency Breakdown Across Serving Platforms (Llama-3-70B): Platform Cold Start Latency Scaling Capacity Resource Efficiency Vanilla Kubernetes + vLLM: 185.4 seconds Slow (1 pod / min) 1.0x (Baseline) Ray Serve (Actor Replicas): 68.2 seconds Moderate 1.4x Triton + Knative Serverless: 54.6 seconds Moderate 1.6x DeepServe (FLOWSERVE Engine): 1.8 seconds 64 instances in 2s 3.2x Cluster Goodput

1. Sub-Second Cold-Start Performance

Under severe traffic bursts, DeepServe’s pre-warmed executors and NPU-Fork mechanism slash cold-start initialization latency by over 98%:

  • Spinning up 1 instance: 1.2 seconds.
  • Scaling in parallel to 16 instances: 1.5 seconds.
  • Scaling in parallel to 64 instances: 1.9 seconds.

In comparison, vanilla Kubernetes requires more than 3 minutes to launch 16 GPU pods due to image pulling, weight transfer, and PyTorch runtime initialization overhead.

2. Cluster Goodput and Cost Efficiency

By combining hybrid prefill-decode scheduling with dynamic NPU-Fork scaling, DeepServe achieves:

  • 3.2x higher overall cluster goodput (percentage of requests meeting strict SLO deadlines) under real-world Azure and Huawei Cloud production traces.
  • 68% reduction in total cost of ownership (TCO) compared to static over-provisioning.
  • Near-zero performance interference between concurrent fine-tuning and interactive inference tasks.
Cluster Utilization Under Day-Long Production Trace: Utilization % 100│ DeepServe Dynamic Fleet 80│ ┌───────────────┐ (Averages 78% Utilization) 60│ ┌───────┘ └───────┐ 40│ ┌───┘ └───┐ 20│───┘ └─── Static Over-Provisioning Baseline 0└───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴───┴─── (Averages 22% Utilization!) 00:00 06:00 12:00 18:00

Production Deployment Playbook: Kubernetes, FlowServe, and Multi-Tenant Quotas

Deploying DeepServe in production cloud environments involves setting up pre-warmed worker pools, configuring host IPC sharing for NPU-Fork, and establishing Prometheus SLO monitoring.

Production Serverless Architecture: ┌────────────────────────────────────────────────────────────────────────┐ | Kubernetes Cloud Cluster | | | | ┌──────────────────────────────────────────────────────────────────┐ | | │ DeepServe API Gateway & Router │ | | │ • Rate Limiting, Token Authentication, Tenant Quota Management │ | | │ • Decomposes Requests into Request-Job-Task DAGs │ | | └──────────────────────────────────┬───────────────────────────────┘ | | │ | | ▼ | | ┌──────────────────────────────────────────────────────────────────┐ | | │ FLOWSERVE Global Cluster Scheduler │ | | │ • Evaluates PD-Disaggregated vs. PD-Colocated Placement │ | | │ • Issues NPU-Fork IPC Direct Memory Commands │ | | └──────────────────────────────────┬───────────────────────────────┘ | | │ | | ▼ | | ┌──────────────────────────────────────────────────────────────────┐ | | │ Pre-Warmed Task Executor (TE) Pods │ | | │ • Pod 0: Prefill Worker Pool (TP=8) │ | | │ • Pod 1: Decode Worker Pool (TP=2) │ | | │ • Pod 2..N: Elastic Hybrid Workers (Scaled via NPU-Fork) │ | | └──────────────────────────────────┬───────────────────────────────┘ | | │ | | ▼ | | ┌──────────────────────────────────────────────────────────────────┐ | | │ Prometheus Exporter & Metrics Collector │ | | │ • deepserve_cold_start_duration_ms │ | | │ • deepserve_active_task_executors │ | | │ • deepserve_slo_compliance_ratio │ | | └──────────────────────────────────────────────────────────────────┘ | └────────────────────────────────────────────────────────────────────────┘

1. Kubernetes Resource Manifest for Task Executors

Ensure that your Kubernetes deployment manifest enables host IPC and shared memory volume mounts required for zero-copy NPU-Fork cloning:

apiVersion: apps/v1 kind: Deployment metadata: name: deepserve-task-executor namespace: ai-serverless spec: replicas: 8 # Base pre-warmed pool selector: matchLabels: app: deepserve-te template: metadata: labels: app: deepserve-te spec: hostIPC: true # Essential for zero-copy NPU-Fork memory sharing containers: - name: flowserve-worker image: ghcr.io/huaweicloud/deepserve-flowserve:latest resources: limits: huawei.com/Ascend910: '8' # Or nvidia.com/gpu: "8" memory: 512Gi cpu: '64' volumeMounts: - mountPath: /dev/shm name: dshm env: - name: DEEPSERVE_ENABLE_NPU_FORK value: '1' - name: DEEPSERVE_PREWARMED_POOL_SIZE value: '16' - name: DEEPSERVE_DRAM_PRELOAD_PATH value: '/mnt/dram_models' volumes: - name: dshm emptyDir: medium: Memory

2. Dispatching Requests with Serverless Priority Headers

Clients querying DeepServe can supply routing hints to help the FLOWSERVE scheduler assign optimal execution tasks:

import requests api_url = "http://deepserve-gateway.ai-serverless.svc.cluster.local:8000/v1/chat/completions" headers = { "Content-Type": "application/json", "x-deepserve-tier": "interactive", "x-deepserve-priority": "high", "x-slo-ttft-ms": "250", "x-slo-tpot-ms": "30" } payload = { "model": "meta-llama/Llama-3-70B-Instruct", "messages": [ {"role": "system", "content": "You are an autonomous cloud infrastructure engineer."}, {"role": "user", "content": "Analyze the attached cluster telemetry and recommend scaling policies."} ], "temperature": 0.2 } response = requests.post(api_url, headers=headers, json=payload) print("Response Status:", response.status_code) print("Content:", response.json()["choices"][0]["message"]["content"][:200])

For teams designing scalable document intelligence pipelines, connecting this inference layer to aiFetchly API Integration or managing enterprise workflows through aiFetchly Web Scraping allows extracting web documents and querying serverless LLM endpoints with minimal GPU infrastructure costs.

Frequently Asked Questions (FAQ)

What is DeepServe and how does it differ from vanilla vLLM?

DeepServe is an open-source serverless AI serving platform developed by Peking University and Huawei Cloud (USENIX ATC 2025). While vanilla vLLM operates as a monolithic, static inference daemon requiring permanent GPU allocation, DeepServe decomposes workloads into a hierarchical Request-Job-Task DAG, uses a microkernel engine (FLOWSERVE), and supports sub-second auto-scaling. It scales from zero to 64 accelerator instances in under 2 seconds, reducing idle infrastructure costs by up to 68%.

What is NPU-Fork and how does it achieve sub-second cold starts?

NPU-Fork is an accelerator process cloning mechanism analogous to the UNIX fork() system call. Instead of loading model weights from disk or host memory over PCIe every time a new instance spins up, a parent template worker holds model weights in accelerator HBM. When scaling out, child workers fork from the parent and map its read-only model weight address space directly via IPC. Only dynamic runtime state (KV cache) is allocated fresh, allowing new instances to become operational in under 1.5 seconds.

How does DeepServe’s hybrid scheduling differ from pure PD disaggregation?

Pure Prefill-Decode (PD) disaggregation (used in systems like DistServe) strictly divides hardware into dedicated prefill nodes and dedicated decode nodes. DeepServe proves that pure disaggregation suffers from pool starvation when prompt/decode ratios fluctuate. DeepServe introduces a three-tier architecture: dedicated prefill workers, dedicated decode workers, and an elastic hybrid colocated pool that dynamically absorbs burst variance.

Can DeepServe be deployed on NVIDIA GPUs as well as Huawei Ascend NPUs?

Yes. Although DeepServe was hardened in production on large Huawei Ascend NPU clusters, its Request-Job-Task abstraction, microkernel architecture, and memory-sharing mechanisms are hardware-agnostic and fully compatible with NVIDIA GPU clusters supporting CUDA IPC and Unified Virtual Addressing (UVA).

How does DeepServe support post-training and fine-tuning alongside inference?

Because DeepServe abstracts execution into fine-grained atomic tasks, it can interleave low-priority background fine-tuning tasks (such as LoRA gradient backpropagation) with high-priority interactive inference tasks on the same accelerator pool, automatically preempting fine-tuning tasks when an inference surge arrives.

Where can I read the DeepServe research paper and access the codebase?

The DeepServe research paper is published in the Proceedings of the USENIX Annual Technical Conference 2025 (DEEPSERVE: Serverless Large Language Model Serving at Scale, arXiv:2501.14417). The official repository and architectural documentation are hosted by Huawei Cloud and Peking University.

To learn more about optimizing data retrieval and AI agents, consult our aiFetchly Documentation or review our enterprise aiFetchly Pricing plans.