opentelallm-servingdistributed-systemsosdi-2026hpcslurmcrdttutorial

OpenTela Guide: Decentralized Heterogeneous LLM Serving

OpenTela Architecture: Decentralized Heterogeneous LLM Serving Overlay

The Sovereign AI Imperative and the HPC Serving Dilemma

In modern enterprise automation, automated market intelligence pipelines, and desktop research copilots powered by aiFetchly, foundation models are increasingly embedded into mission-critical workflows. Whether extracting thousands of structured documents via real-time data fetching, parsing complex web schemas using automated web scraping, or orchestrating autonomous tasks through custom API integrations, organizations face growing regulatory, privacy, and economic pressures to deploy models on infrastructure they directly own and govern.

This shift has catalyzed the global Sovereign AI movement, where academic consortia, national research centers, and enterprise engineering teams seek to run large language models (LLMs) on their private supercomputing facilities.

However, attempting to serve interactive, low-latency LLM inference on high-performance computing (HPC) hardware exposes a fundamental architectural mismatch: HPC clusters were designed for batch-oriented scientific simulations, not interactive cloud microservices.

The Fundamental Architectural Mismatch: ┌────────────────────────────────────────┐ ┌────────────────────────────────────────┐ │ Cloud-Native LLM Serving (Kubernetes) │ │ High-Performance Computing (HPC/Slurm) │ ├────────────────────────────────────────┤ ├────────────────────────────────────────┤ │ • Always-on microservice pods │ VS. │ • Ephemeral batch job allocations │ │ • Centralized etcd / kube-apiserver │ │ • Rigid Slurm queue management │ │ • Software-defined network overlay │ │ • Strict air-gapped node isolation │ │ • Full root / privileged container access│ │ • Non-root user space only │ │ • Homogeneous hyperscaler GPU pools │ │ • Heavily fragmented GPU architectures │ └────────────────────────────────────────┘ └────────────────────────────────────────┘

Modern inference engines (such as vLLM, SGLang, and TensorRT-LLM) excel at maximizing token throughput on raw GPUs. Yet, they rely entirely on an external cloud control plane (like Kubernetes) for service discovery, dynamic request routing, health probing, and load balancing. In HPC supercomputers—such as the Swiss National Supercomputing Centre (CSCS) Alps infrastructure or national university labs—these cloud primitives simply do not exist.

The Four Fundamental Barriers in HPC LLM Serving

When systems engineers attempt to adapt existing HPC supercomputer allocations for production LLM serving, they encounter four critical barriers:

The Four Core Challenges in HPC LLM Serving: ┌────────────────────────────────────────────────────────────────────────┐ │ 1. Non-Root User Space Constraint │ │ • Users lack administrative/root privileges on HPC clusters. │ │ • Impossible to install Kubernetes kubelet daemons, wire up CNI │ │ plugins, modify iptables, or configure kernel-level wireguards. │ ├────────────────────────────────────────────────────────────────────────┤ │ 2. Transient Allocations & Job Queue Latency │ │ • HPC compute nodes are scheduled as batch jobs with hard wall-times. │ │ • Nodes are terminated when time limits expire or preempted for jobs. │ │ • Cold-start queue wait times (minutes to hours) break interactive SLAs.│ ├────────────────────────────────────────────────────────────────────────┤ │ 3. Network Isolation & Private Ingress Inaccessibility │ │ • High-performance compute nodes lack public IP addresses. │ │ • Restrictive firewalls block direct inbound TCP/HTTP connections. │ │ • Traditional HTTP reverse proxies cannot route to compute workers. │ ├────────────────────────────────────────────────────────────────────────┤ │ 4. Severe Cross-Cluster Hardware Heterogeneity │ │ • Clusters span heterogeneous GPU hardware: NVIDIA GH200, H100, │ │ A100, V100, RTX 4090s, and AMD Instinct MI300X. │ │ • Fragmented memory capacities and interconnects require fine-tuned │ │ tensor parallelism degrees and engine placements per device tier. │ └────────────────────────────────────────────────────────────────────────┘

Because of these barriers, organizations historically faced an unpalatable dilemma: either spend millions of dollars migrating workloads to commercial cloud providers, or maintain brittle, handcrafted SSH port-forwarding scripts that break whenever a Slurm allocation expires.

OpenTela System Architecture: The User-Space Overlay

To resolve this impasse, a multi-institutional team from ETH Zurich, University of Cambridge, EPFL, MIT, HKUST, and CSCS introduced OpenTela: Unifying Decentralized Computing Resources for Heterogeneous LLM Serving at USENIX OSDI 2026 (Operational Systems, pages 1821–1838).

OpenTela is a lightweight user-space orchestration overlay that turns fragmented, multi-institutional HPC clusters, Kubernetes environments, and bare-metal nodes into a unified, federated LLM serving platform. OpenTela operates entirely in user space: it requires zero root privileges, zero cluster reconfigurations, and zero kernel network drivers.

OpenTela System Architecture: ┌───────────────────────────────────────────────────────────────────────────────────────┐ │ PUBLIC CLIENT INGRESS (OpenAI-Compatible REST Gateway) │ │ • Receives /v1/chat/completions requests from web apps, APIs & desktop copilots │ │ • Local CRDT Read Replica: Resolves healthiest, lowest-latency backend node instantly │ └───────────────────────────────────┬───────────────────────────────────────────────────┘ │ Reverse gRPC / WebSocket Tunneling ▼ ┌───────────────────────────────────────────────────────────────────────────────────────┐ │ P2P GOSSIP NETWORK (Monotonic CRDT Registry) │ │ • Coordinator-Free Service Discovery & Health Monitoring │ │ • Peer-to-peer state propagation across NAT firewalls and administrative domains │ └───────────────────────────────────┬───────────────────────────────────────────────────┘ │ ┌──────────────────────────┼──────────────────────────┐ ▼ ▼ ▼ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ HPC CLUSTER │ │ KUBERNETES │ │ BARE-METAL FARM │ │ (CSCS Alps) │ │ CLOUD CLUSTER │ │ (Workstations) │ │ • Slurm Batch │ │ • Pod Daemon │ │ • Standalone │ │ Allocations │ │ • Always-On │ │ Node Daemon │ │ • NVIDIA GH200 │ │ • NVIDIA A100 │ │ • NVIDIA RTX │ │ • Transient │ │ • Elastic Auto- │ │ • Local Testing │ │ Node Spinning │ │ scaling │ │ & Prototyping │ └─────────────────┘ └─────────────────┘ └─────────────────┘

OpenTela comprises five interconnected modules:

  1. API Ingress Nodes: User-facing gateway nodes positioned at publicly reachable IP addresses that terminate TLS, validate authentication tokens, and expose standard OpenAI-compatible endpoints (/v1/chat/completions, /v1/embeddings).
  2. Serving Node Daemons: Lightweight Python binaries launched inside user jobs (e.g., Slurm batch scripts or Kubernetes pods) that supervise underlying inference engines (vLLM, SGLang) and establish reverse communication channels.
  3. CRDT Gossip Registry: A decentralized, peer-to-peer state registry that maintains a synchronized catalog of active compute nodes and model deployments without a central coordinator.
  4. Heterogeneity-Aware Scheduler: An intelligent placement engine that maps incoming requests and model weights to the optimal GPU tier based on memory footprints, interconnect bandwidths, and service-level objectives (SLOs).
  5. Model Spinning Lifecycle Manager: An automated batch job orchestrator that dynamically submits and drains Slurm job allocations via the FirecREST REST API.

Coordinator-Free Discovery: The CRDT-Backed Gossip Network

Traditional service meshes (such as Consul, etcd, or ZooKeeper) rely on strongly consistent quorum protocols (like Raft or Paxos). In federated cross-institutional environments, strict quorums fail completely: network partitions across institutional firewalls stall elections, and compute nodes behind strict NAT gateways cannot participate in voting loops.

OpenTela solves this by framing service discovery as an eventually consistent Conflict-Free Replicated Data Type (CRDT) governed by an epidemic gossip protocol.

The Node Lifecycle State Machine: ┌───────────┐ Heartbeat OK ┌───────────┐ │ JOIN │ ─────────────────────► │ SERVE │ └───────────┘ └───────────┘ │ Heartbeat Timeout │ Drain / Preempt ▼ ┌───────────┐ Tombstone Epoch ┌───────────┐ │ LEFT │ ◄───────────────────── │ DOWN │ └───────────┘ └───────────┘

Monotonic State Lattice Formulation

Every OpenTela node maintains a local replica of the global state lattice. The state of node $n$ is defined by a tuple:

$$S_n = \langle \text{node_id}, \text{state}, \text{generation}, \text{heartbeat_ts}, \text{models}, \text{hardware_caps} \rangle$$

Where:

  • $\text{state} \in {\text{JOIN}, \text{SERVE}, \text{DOWN}, \text{LEFT}}$ progresses strictly monotonically.
  • $\text{generation} \in \mathbb{N}$ is a monotonically increasing integer incremented whenever a node restarts.
  • $\text{heartbeat_ts}$ is the local physical timestamp of the latest health probe.
  • $\text{models}$ is the set of hosted model identifiers and quantized formats.
  • $\text{hardware_caps}$ encodes GPU architecture, VRAM capacity, and tensor parallelism configurations.

When two node replicas $S_A$ and $S_B$ exchange state via gossip, the merge operator $\sqcup$ computes the least upper bound (join-semilattice):

CRDT Merge Logic: If generation(A) > generation(B): Merge Result = State(A) Else if generation(B) > generation(A): Merge Result = State(B) Else: # Same generation: prioritize advanced lifecycle state State Priority: LEFT > DOWN > SERVE > JOIN Merge Result = max_state(State(A), State(B)) heartbeat_ts = max(heartbeat_ts(A), heartbeat_ts(B))

Because the state transition space forms a monotonic lattice, concurrent gossip exchanges are guaranteed to converge deterministically ($A \sqcup B = B \sqcup A$), irrespective of packet reordering, network drops, or message duplication across WAN links.

Every ingress node holds a full local replica of this CRDT registry. When a client submits an inference request, the ingress gateway evaluates routing candidates locally from memory in under 0.5 milliseconds, without incurring cross-network coordination hops.

Heterogeneity-Aware Scheduling & Model Placement

In a federated computing environment, GPU capacity varies wildly across clusters. For instance, CSCS Alps provides high-density NVIDIA GH200 nodes (96 GB HBM3 + 480 GB unified LPDDR5X memory), while university clusters host standard NVIDIA A100 (80 GB) nodes, and research labs contribute workstations with consumer RTX 4090 (24 GB) GPUs.

Placing models arbitrarily across these resources results in extreme inefficiency. Running a compact 8B parameter model on a 96 GB GH200 node squanders valuable high-bandwidth memory, while attempting to run a 70B parameter model across multiple PCIe-connected RTX 4090s triggers catastrophic pipeline stalls.

OpenTela’s Heterogeneity-Aware Scheduler formalizes model placement as a constrained multi-tier optimization problem:

Hardware Tier Classification in OpenTela: ┌──────────────┬────────────────────────┬─────────────────────┬────────────────────────┐ │ Tier Class │ Representative Devices │ Interconnect │ Target Model Workloads │ ├──────────────┼────────────────────────┼─────────────────────┼────────────────────────┤ │ Tier 1 (HPC) │ NVIDIA GH200 / H100 │ NVLink (900 GB/s) │ Frontier 70B+ FP16, │ │ │ (80GB – 96GB HBM3) │ Infiniband NDR │ MoE, Long-Context (>32k)│ ├──────────────┼────────────────────────┼─────────────────────┼────────────────────────┤ │ Tier 2 (Mid) │ NVIDIA A100 / L40S │ NVLink / PCIe Gen4 │ Mid-Size 13B–34B FP16, │ │ │ (40GB – 80GB HBM2e) │ 100GbE RoCE │ Quantized 70B (AWQ/INT4)│ ├──────────────┼────────────────────────┼─────────────────────┼────────────────────────┤ │ Tier 3 (Edge)│ NVIDIA RTX 4090 / 3090 │ PCIe Gen4 (32 GB/s) │ Compact 3B–8B INT4, │ │ │ (16GB – 24GB GDDR6X) │ Standard 10GbE LAN │ Embedding Models (BERT)│ └──────────────┴────────────────────────┴─────────────────────┴────────────────────────┘

The placement engine profiles two primary metrics before scheduling a model deployment:

  1. Memory-to-Parameter Ratio: $\frac{VRAM_{avail}}{Weights_{bytes} + KV_{headroom}}$. If the ratio is below 1.25, the engine enforces tensor parallelism across adjacent NVLink devices or selects an aggressively quantized checkpoint (e.g., GPTQ or AWQ).
  2. Communication-to-Computation Bound: When a model requires multi-node pipeline parallelism, OpenTela’s topology filter verifies that all member nodes reside within the same InfiniBand or high-speed RoCE fabric, strictly rejecting cross-WAN or PCIe-bridged pipeline splits.

Dynamic Model Spinning: Bridging Batch Jobs to Interactive Traffic

A central operational innovation in OpenTela is Model Spinning.

In typical HPC facilities, users submit jobs that wait in queue until resources become available. If a model was only served during fixed batch reservations, interactive availability would be completely unpredictable.

OpenTela bridges this divide by integrating with HPC REST interfaces—such as the FirecREST API on CSCS Alps. The Model Spinning controller continuously monitors incoming request queues. When traffic surges for a specific model:

  1. Automated Slurm Submission: The controller issues a pre-templated Slurm allocation request specifying optimal node shapes, job wall-times (e.g., 4 to 8 hours), and GPU counts.
  2. Fast Weight Staging: Slurm compute nodes pull model weights from a shared Lustre/GPFS parallel filesystem or local SSD scratch caches, launching an optimized vLLM or SGLang worker in under 45 seconds.
  3. Gossip Self-Registration: The newly spawned OpenTela daemon broadcasts a JOIN message into the gossip overlay. Within two gossip rounds (<1.5 seconds), ingress gateways recognize the worker and start routing inference traffic.
  4. Graceful Job Drainage: Fifteen minutes prior to Slurm job wall-time expiration, the worker broadcasts a DOWN transition. Ingress nodes immediately divert new requests to alternative instances while allowing in-flight streaming responses to terminate cleanly.
The Model Spinning Lifecycle: Traffic Surge ──► FirecREST API ──► Slurm Queue Allocation (CSCS Alps) │ ▼ Compute Node Launches OpenTela Daemon │ ▼ vLLM / SGLang Engine Initialized (<45s) │ ▼ P2P Gossip Registration (State: SERVE) │ ▼ Ingress Gateways Route Inference Traffic │ ▼ Wall-Time - 15m ──► Broadcast State: DOWN │ ▼ In-Flight Drain ──► Slurm Job Clean Exit

Implementation Blueprint: Building a CRDT Node Registry in Python

Below is a complete, runnable Python implementation demonstrating the core mechanics of OpenTela’s CRDT-based monotonic lattice registry, node state lifecycle tracking, and gossip merge resolution.

""" opentela_registry.py Core implementation of OpenTela's CRDT Monotonic Lattice Registry: - Node lifecycle states: JOIN, SERVE, DOWN, LEFT - Monotonic state-based CRDT merge operator (join-semilattice) - Peer-to-peer gossip propagation simulation - Candidate node selection for OpenAI-compatible ingress routing """ from dataclasses import dataclass, field from enum import IntEnum import time from typing import Dict, List, Optional, Set class NodeState(IntEnum): JOIN = 1 SERVE = 2 DOWN = 3 LEFT = 4 @dataclass class HardwareProfile: gpu_model: str num_gpus: int vram_gb_per_gpu: float interconnect: str # e.g., "NVLink", "PCIe-Gen4", "Infiniband-NDR" @dataclass class NodeRecord: node_id: str state: NodeState generation: int heartbeat_ts: float models: Set[str] = field(default_factory=set) hardware: Optional[HardwareProfile] = None def copy(self) -> 'NodeRecord': return NodeRecord( node_id=self.node_id, state=self.state, generation=self.generation, heartbeat_ts=self.heartbeat_ts, models=set(self.models), hardware=self.hardware, ) class MonotonicCRDTRegistry: def __init__(self, local_node_id: str): self.local_node_id = local_node_id # In-memory lattice storage: node_id -> NodeRecord self.registry: Dict[str, NodeRecord] = {} def register_or_update_local( self, state: NodeState, generation: int, models: Set[str], hardware: HardwareProfile, ) -> None: """Updates the local node's own presence record.""" self.registry[self.local_node_id] = NodeRecord( node_id=self.local_node_id, state=state, generation=generation, heartbeat_ts=time.time(), models=models, hardware=hardware, ) def merge_remote_record(self, remote: NodeRecord) -> bool: """ State-based CRDT merge operator (join-semilattice). Returns True if local state was updated, False otherwise. """ local = self.registry.get(remote.node_id) if local is None: self.registry[remote.node_id] = remote.copy() return True updated = False # Rule 1: Higher generation strictly dominates older generation if remote.generation > local.generation: self.registry[remote.node_id] = remote.copy() return True elif remote.generation < local.generation: return False # Discard stale generation # Rule 2: Same generation - state advances monotonically (LEFT > DOWN > SERVE > JOIN) if remote.state > local.state: local.state = remote.state updated = True # Rule 3: Merge heartbeat timestamp if remote.heartbeat_ts > local.heartbeat_ts: local.heartbeat_ts = remote.heartbeat_ts updated = True # Rule 4: Set union of advertised models if not remote.models.issubset(local.models): local.models.update(remote.models) updated = True return updated def get_serving_candidates(self, model_name: str) -> List[NodeRecord]: """ Returns all healthy nodes currently in the SERVE state hosting the target model. """ current_time = time.time() timeout_threshold_sec = 15.0 candidates = [] for record in self.registry.values(): if record.state == NodeState.SERVE and model_name in record.models: # Health probe: ensure node hasn't timed out if (current_time - record.heartbeat_ts) < timeout_threshold_sec: candidates.append(record) return candidates class IngressRouter: def __init__(self, registry: MonotonicCRDTRegistry): self.registry = registry def resolve_endpoint(self, model_name: str) -> Optional[NodeRecord]: """ Selects the optimal hardware node for a requested model. Prefers NVLink/HPC nodes for large models, falls back to edge/commodity. """ candidates = self.registry.get_serving_candidates(model_name) if not candidates: return None # Sort candidates: prefer nodes with highest VRAM and NVLink interconnect def rank_node(node: NodeRecord) -> float: score = 0.0 if node.hardware: score += node.hardware.vram_gb_per_gpu * node.hardware.num_gpus if "NVLink" in node.hardware.interconnect: score += 1000.0 return score candidates.sort(key=rank_node, reverse=True) return candidates[0] if __name__ == "__main__": print("[OpenTela] Initializing Decentralized CRDT Serving Overlay...") # Node 1: Ingress Gateway (Public IP) ingress_registry = MonotonicCRDTRegistry("ingress-zurich-01") # Node 2: CSCS Alps Supercomputer Slurm Worker (Private Network) alps_hw = HardwareProfile( gpu_model="NVIDIA GH200", num_gpus=4, vram_gb_per_gpu=96.0, interconnect="NVLink-C2C", ) alps_node = NodeRecord( node_id="cscs-alps-nid00412", state=NodeState.SERVE, generation=1, heartbeat_ts=time.time(), models={"meta-llama/Llama-3-70B-Instruct", "mistralai/Mixtral-8x22B"}, hardware=alps_hw, ) # Node 3: Academic Kubernetes Worker (Tier 2 Cloud) k8s_hw = HardwareProfile( gpu_model="NVIDIA A100", num_gpus=2, vram_gb_per_gpu=80.0, interconnect="PCIe-Gen4", ) k8s_node = NodeRecord( node_id="eth-k8s-worker-gpu07", state=NodeState.SERVE, generation=1, heartbeat_ts=time.time(), models={"meta-llama/Llama-3-70B-Instruct", "Qwen/Qwen2.5-7B"}, hardware=k8s_hw, ) # Ingress receives gossip updates ingress_registry.merge_remote_record(alps_node) ingress_registry.merge_remote_record(k8s_node) router = IngressRouter(ingress_registry) target_model = "meta-llama/Llama-3-70B-Instruct" selected = router.resolve_endpoint(target_model) if selected: print(f"[Router Match] Request for '{target_model}' routed to:") print(f" -> Node ID: {selected.node_id}") print(f" -> Hardware: {selected.hardware.gpu_model} ({selected.hardware.interconnect})") print(f" -> State: {selected.state.name} | Total VRAM: {selected.hardware.vram_gb_per_gpu * selected.hardware.num_gpus} GB")

22-Month Production Trace Analysis & Benchmarks

In the USENIX OSDI 2026 operational paper, the authors presented extensive empirical findings gathered from the Swiss AI Initiative production deployment, which powered generative research across ETH Zurich, EPFL, and the Swiss National Supercomputing Centre (CSCS).

OpenTela Production Deployment Statistics (22 Months): ┌────────────────────────────────────────────────────────┐ │ Metric Production Scale │ ├────────────────────────────────────────────────────────┤ │ Total Requests Served 13.2 Million Requests │ │ Total Generated Tokens 15.4 Billion Tokens │ │ Concurrent Models Hosted 142 Unique Models │ │ Active Research Users 1,000+ Researchers │ │ Cluster Domains Federated CSCS Alps, ETH, EPFL │ │ Average Routing Latency < 3.8 Milliseconds │ │ Ingress Uptime Availability 99.98% Service Uptime │ └────────────────────────────────────────────────────────┘

Key Performance Findings

  1. Routing and Ingress Overhead: Thanks to local CRDT in-memory lookups, the total routing decision overhead at the ingress node averaged under 3.8 milliseconds per request—completely negligible compared to standard LLM Time-to-First-Token (TTFT) metrics (typically 150ms to 400ms).
  2. Resilience to Slurm Preemptions: During a 90-day stress analysis, over 1,200 Slurm batch jobs expired or were preempted while serving traffic. Thanks to the 15-minute DOWN transition draining protocol, 99.94% of in-flight user conversations completed without interruption, while subsequent prompt requests were automatically re-routed to surviving nodes within 1.2 seconds.
  3. GPU Utilization and Cost Savings: Compared to maintaining static, dedicated cloud instances on AWS or GCP, federating unused opportunistic allocations on CSCS Alps and institutional university clusters slashed operational compute expenditure by over 78%, while unlocking access to frontier 70B+ parameter models on sovereign hardware.

Production Engineering Runbook: Deploying OpenTela on Slurm & Kubernetes

To launch your own federated LLM serving overlay across Slurm and Kubernetes, follow this structured engineering checklist:

OpenTela Deployment Workflow: 1. Deploy Public Ingress Node ├── Host on a VM with public IP & valid SSL certificate (e.g., Let's Encrypt). ├── Run: opentela-ingress --port 443 --crdt-port 9090 --api-key-auth └── Expose OpenAI-compatible REST API endpoints at /v1. 2. Launch Slurm Compute Worker (CSCS Alps / Supercomputer) ├── Create batch launch script (model_serve.slurm). ├── Configure vLLM/SGLang with model weights stored on parallel filesystem. └── Submit batch job: sbatch model_serve.slurm 3. Connect Peer-to-Peer Gossip Overlay └── Slurm worker initiates reverse gRPC tunnel to Ingress gossip IP.

Slurm Batch Script Example (model_serve.slurm)

#!/bin/bash #SBATCH --job-name=opentela-llama3 #SBATCH --nodes=1 #SBATCH --gpus=4 #SBATCH --time=04:00:00 #SBATCH --partition=gh200 # Load modules and activate user environment module load cuda/12.4 source ~/miniconda3/bin/activate opentela-env # Launch vLLM engine in user space (no root required) python -m vllm.entrypoints.openai.api_server \ --model /scratch/shared/models/meta-llama-3-70b-instruct \ --tensor-parallel-size 4 \ --port 8000 & VLLM_PID=$! # Launch OpenTela User-Space Node Daemon # Connects reverse gRPC tunnel back to the public Ingress node opentela-node \ --node-id cscs-alps-$SLURM_JOB_ID \ --ingress-gossip-addr 198.51.100.25:9090 \ --backend-url http://127.0.0.1:8000 \ --model meta-llama/Llama-3-70B-Instruct \ --hardware-profile gh200_4x \ --drain-before-timeout-sec 900 wait $VLLM_PID

Once registered, external applications, multi-agent orchestrators, and enterprise pipelines can query the unified endpoint through developer documentation. For enterprises evaluating private infrastructure migrations, explore our flexible pricing plans for custom architecture consultations.

Frequently Asked Questions (FAQ)

Does OpenTela require root privileges or sudo on the HPC cluster?

No. OpenTela is engineered strictly as a user-space overlay. Both the OpenTela node daemon and the underlying serving engines (vLLM, SGLang) run under standard unprivileged user accounts, using user-space networking and outbound reverse tunnels that comply with standard HPC security rules.

How does OpenTela traverse strict HPC firewalls without public IPs?

Compute nodes inside HPC clusters typically block inbound connections but allow outbound HTTPS/gRPC connections. OpenTela nodes initiate an outbound reverse WebSocket/gRPC tunnel to the public Ingress node upon startup. All client inference requests are multiplexed through this established outbound tunnel, requiring zero firewall reconfiguration.

Can OpenTela manage both Slurm and Kubernetes simultaneously?

Yes. OpenTela is platform-agnostic. Ingress gateways treat Slurm batch jobs, Kubernetes pods, and bare-metal servers as peer nodes in the same CRDT gossip lattice, routing requests seamlessly across heterogeneous hosting environments.

What serving backends are supported?

OpenTela works out-of-the-box with any serving engine that exposes an HTTP or gRPC API, including vLLM, SGLang, HuggingFace TGI, and TensorRT-LLM.