Agentic WorkflowsCloud OrchestrationCompound AI SystemsSystems EngineeringOSDI 2026

Murakkab Guide: Agentic Workflow Cloud Orchestration

Murakkab Architecture: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms

Introduction: The Systems Crisis of Compound AI Agent Workflows

The architectural center of gravity in enterprise artificial intelligence has definitively transitioned from single-prompt conversational language models to Compound AI Systems and Agentic Workflows. Instead of expecting a single massive neural network to solve every problem in one monolithic completion, modern AI architectures decompose complex tasks into coordinated graphs of smaller, specialized operations. Autonomous software copilots, multi-hop financial analysts, scientific research synthesizers, and enterprise web bots coordinate diverse models—frontier Large Language Models (LLMs), lightweight Small Language Models (SLMs), embedding models, code verifiers—interleaved with external tool calls, database lookups, and API invocations.

In enterprise data pipelines powered by automated web scraping infrastructure, real-time data fetching, and robust API integration backends, these compound agentic workflows represent the overwhelming majority of daily production traffic.

However, hosting and orchestrating agentic workflows across cloud infrastructure introduces a fundamental systems crisis:

  1. The Opaque Client-Side Trap: Existing agent frameworks—such as LangChain, AutoGen, CrewAI, and Semantic Kernel—implement workflows as client-side Python scripts. These frameworks tightly couple high-level business logic to hardcoded model checkpoints (e.g., hardcoding gpt-4o or claude-3-5-sonnet across every node) and hardware instances.
  2. Cloud Schedulers Are Structurally Blind: To cloud providers (such as Microsoft Azure, AWS, and GCP), an agentic workflow is completely invisible. Cloud inference gateways see only a stream of disjoint, isolated HTTP requests. The cloud has zero visibility into the workflow’s Directed Acyclic Graph (DAG), data dependencies, loops, retries, or end-to-end Service Level Objectives (SLOs).
  3. Massive Economic and Energy Over-Provisioning: Because developers cannot predict runtime latency fluctuations or downstream tool delays, they default to worst-case over-provisioning: utilizing expensive, high-power 70B+ models for trivial intermediate steps like entity extraction, query routing, or JSON formatting. When an upstream tool (such as an external web search or sandbox compiler) experiences unexpected latency, rigid downstream pipelines miss their end-to-end latency deadlines entirely.

Enter Murakkab (derived from the Arabic word for compound or composite), a landmark systems paper presented at the 2026 USENIX Symposium on Operating Systems Design and Implementation (OSDI 2026) by Gohar Irfan Chaudhry, Esha Choukse, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Adam Belay, and Ricardo Bianchini, representing a premier research collaboration between MIT CSAIL and Microsoft Azure.

Murakkab introduces the first resource-efficient cloud serving system built specifically for multi-model agentic workflows. By introducing a declarative abstraction that decouples workflow specifications from physical execution configurations, utilizing a profile-guided optimizer (PGO) to navigate multi-dimensional Pareto frontiers across accuracy, latency, energy, and cost, and deploying an adaptive runtime with dynamic real-time reconfiguration, Murakkab slashes cloud serving costs by up to 4.3x, reduces GPU consumption by 2.8x, and decreases datacenter energy usage by 3.7x while strictly preserving user-defined SLOs.

In this comprehensive technical guide, we examine the structural inefficiencies of today’s agentic deployments, dissect the declarative abstractions of Murakkab, explore its profile-guided optimizer, walk through a complete Python/PyTorch reference implementation, and discuss the architectural implications for production compound AI systems in 2026.

Anatomy of the Opaque Workflow Problem in Cloud Platforms

To understand why conventional cloud inference serving fails on compound AI systems, we must analyze the interaction between modern agent frameworks and cloud resource managers.

The Lifecycle of an Agentic Workflow DAG

An agentic workflow is formally characterized as an execution graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$, where vertices $v \in \mathcal{V}$ represent operational tasks (LLM prompts, tool executions, parsers, evaluators) and directed edges $e = (u, v) \in \mathcal{E}$ represent dataflow dependencies. Control flow features conditional branching, iterative self-correction loops, and dynamic sub-task fan-outs.

Consider a representative enterprise customer intelligence workflow:

  • Task $T_1$ (Input Ingestion & Query Classification): Classifies the incoming customer intent into product complaints, feature requests, or technical support queries.
  • Task $T_2$ (External Tool Retrieval): Dispatches automated tool calls to query internal SQL databases and live web documentation.
  • Task $T_3$ (Multi-Document Synthesis): Ingests the retrieved documentation and drafts a comprehensive solution.
  • Task $T_4$ (Code Verification & Formatting): Validates code snippets in a sandbox and structures the final response into typed JSON schema.
+-------------------------------------------------------------------------------+ | REPRESENTATIVE AGENTIC WORKFLOW DAG | +-------------------------------------------------------------------------------+ | | | [Customer Input] ---> Task T1: Intent Classification (Low Compute SLM) | | | | | +---> Task T2: Tool Retrieval (Web / SQL) | | | | | v | | Task T3: Synthesis & Reasoning (Frontier LLM) | | | | | v | | Task T4: Verification & Output (Code / Schema) | | | +-------------------------------------------------------------------------------+ | The Existing Framework Trap: | | •LangChain/AutoGen script hardcodes: T1=GPT-4, T3=GPT-4, T4=GPT-4 | | •Cloud gateway receives: POST /v1/chat (T1), then POST /v1/chat (T3)... | | •Cloud sees 3 isolated requests; has NO visibility into T1->T2->T3->T4 DAG! | +-------------------------------------------------------------------------------+

The Three Structural Flaws of Blind Cloud Schedulers

When cloud platforms treat agentic workflows as uncoordinated sequences of individual requests, three systemic inefficiencies emerge:

1. Inability to Balance Multi-Objective Tradeoffs

An enterprise user cares about end-to-end workflow SLOs:

  • Total workflow wall-clock latency: $T_{\text{total}} \le 2.5\text{ seconds}$
  • Solution quality/accuracy score: $A_{\text{total}} \ge 90%$
  • Cloud operational cost budget: $C_{\text{total}} \le $0.02\text{ per workflow run}$

However, because existing cloud schedulers treat each task as an isolated HTTP request, they cannot optimize across the whole graph. An engineer forced to choose models statically will select the most capable model (e.g., a 70B parameter frontier model) for every step to guarantee accuracy. Yet, experiments in the OSDI 2026 paper show that for Step $T_1$ (intent classification) and Step $T_4$ (schema formatting), an 8B SLM or an INT4-quantized model delivers identical $100%$ accuracy at a fraction of the latency and cost.

2. Cascading Latency Slack Collapse

In compound workflows, external tools introduce non-deterministic latency. In Step $T_2$, a web search or database query might normally take 200ms, but occasionally suffers a 700ms network tail latency spike.

Under static client-side frameworks, because Task $T_3$ is rigidly bound to a heavy model on a specific GPU tier, this 500ms external tool delay directly propagates into the critical path, causing the entire workflow to violate its end-to-end latency SLO. A workflow-aware scheduler, by contrast, could detect the upstream delay and dynamically upgrade Task $T_3$ to a faster model variant or accelerated inference engine to recover the lost slack.

3. Resource Stranding Across Heterogeneous Clusters

Modern cloud datacenters deploy heterogeneous GPU hardware: NVIDIA H100 SXM, A100 80GB, L40S, and modern multi-core CPUs. Different workflow tasks have wildly different resource affinities: prefill-heavy summarization prefers H100 Tensor Cores, while low-concurrency lightweight parsing runs with high energy efficiency on L40S or CPU nodes. Without DAG visibility, cloud schedulers cannot place workflow stages onto their hardware sweet spots, stranding high-cost GPU capacity.

The Murakkab Architecture: Principles and Core Mechanics

Murakkab resolves these challenges by introducing a full-stack, workflow-aware cloud serving architecture. Rather than leaving agentic orchestration to fragmented client-side scripts, Murakkab brings workflow orchestration directly into the cloud platform.

Murakkab’s architecture is organized around three foundational pillars:

  1. Declarative Workflow Abstraction: A standardized programming interface that decouples high-level task specifications and quality constraints from physical model checkpoints and hardware choices.
  2. Profile-Guided Optimizer (PGO): An offline and online optimization engine that models the multi-dimensional Pareto frontier (accuracy, latency, energy, cost) across model families, quantization levels, and hardware targets for each workflow node.
  3. Adaptive Runtime with Dynamic Reconfiguration: A real-time cloud orchestrator that monitors end-to-end latency slack as tasks execute along the DAG, dynamically steering model selection and hardware routing to guarantee SLO compliance at minimum cost.
+-----------------------------------------------------------------------------------------+ | MURAKKAB SYSTEM TOPOLOGY | +-----------------------------------------------------------------------------------------+ | | | [User / Developer] | | | | | v | | +-----------------------------------------------------------------------------------+ | | | DECLARATIVE WORKFLOW SPECIFICATION (Graph DAG + Constraints) | | | | •Defines Tasks, Edges, Dependencies, and End-to-End Latency/Accuracy SLOs | | | +-----------------------------------------------------------------------------------+ | | | | | v | | +-----------------------------------------------------------------------------------+ | | | PROFILE-GUIDED OPTIMIZER (PGO) | | | | •Offline Profiler: Measures (Accuracy, Latency, Energy, Cost) per Task Variant | | | | •Pareto Frontier Solver: Generates Optimal (Model, Precision, HW) Execution Plans | | | +-----------------------------------------------------------------------------------+ | | | | | v | | +-----------------------------------------------------------------------------------+ | | | ADAPTIVE WORKFLOW RUNTIME (Cloud Orchestrator) | | | | •Real-Time Slack Monitor: Tracks upstream execution times & tool delays | | | | •Dynamic Reconfigurator: Re-allocates models & hardware to absorb latency jitter | | | +-----------------------------------------------------------------------------------+ | | | | | +--------------------------+--------------------------+ | | | Stage 1 Route | Stage 2 Route | Stage 3 Route | | v v v | | +--------------------+ +--------------------+ +--------------------+ | | | L40S / CPU Worker | | H100 Tensor Worker | | A100 FP8 Worker | | | | [SLM Classifier] | | [Frontier LLM Syn] | | [Code Verifier] | | | +--------------------+ +--------------------+ +--------------------+ | | | +-----------------------------------------------------------------------------------------+

Deep Dive: Declarative Abstractions & Profile-Guided Optimization

Let us analyze how Murakkab models, profiles, and optimizes agentic workflows.

Declarative Workflow Specification

In Murakkab, developers do not hardcode model names or API keys in application code. Instead, developers declare tasks in terms of semantic intent, data schemas, and quality thresholds:

# Murakkab Declarative Workflow Definition workflow = MurakkabWorkflow( name="CustomerSupportSynthesis", slo_latency_ms=2500, # End-to-end deadline: 2.5 seconds min_accuracy_score=0.92, # Minimum acceptable quality: 92% max_cost_cents=1.5 # Maximum allowable cost per run: 1.5 cents ) t1 = workflow.add_task( name="intent_classification", task_type="classification", input_schema={"query": str}, quality_metric="f1_score" ) t2 = workflow.add_tool( name="kb_retrieval", tool_type="vector_search", depends_on=[t1] ) t3 = workflow.add_task( name="response_synthesis", task_type="generation", depends_on=[t1, t2], quality_metric="bert_score" )

By decoupling what must be computed from how and where it is executed, the cloud platform gains the freedom to select the optimal model size, quantization format, and GPU architecture for each task.

The Profile-Guided Optimizer (PGO)

For each task $v \in \mathcal{V}$, the PGO evaluates a candidate set of execution variants $\mathcal{K}_v$. An execution variant $k = (m, q, h)$ represents a combination of:

  • Model Checkpoint ($m$): e.g., LLaMA-3-8B, Mistral-7B, LLaMA-3-70B, Qwen-2.5-72B.
  • Quantization Precision ($q$): e.g., FP16, FP8, INT4, AWQ.
  • Hardware Target ($h$): e.g., NVIDIA H100, A100, L40S, AMD EPYC CPU.

The PGO profiles each variant $k \in \mathcal{K}_v$ across four fundamental dimensions:

  1. Accuracy ($A_{v,k}$): Evaluated over validation datasets or active telemetry.
  2. Execution Latency ($L_{v,k}$): Measured as prefill + decoding time under representative input lengths.
  3. Energy Footprint ($E_{v,k}$): Measured in joules per execution using datacenter hardware counters.
  4. Cloud Dollar Cost ($C_{v,k}$): Calculated based on hardware hourly leasing rates and execution duration.

The global optimization problem for a workflow DAG with $N$ tasks is formulated as an Integer Linear Program (ILP):

$$\min_{{x_{v,k}}} \sum_{v \in \mathcal{V}} \sum_{k \in \mathcal{K}v} x{v,k} \cdot C_{v,k}$$

subject to:

  • Single Variant Selection: $\sum_{k \in \mathcal{K}v} x{v,k} = 1 \quad \forall v \in \mathcal{V}$
  • End-to-End Latency SLO: $\sum_{v \in \mathcal{P}{\text{crit}}} \sum{k \in \mathcal{K}v} x{v,k} \cdot L_{v,k} \le \text{SLO}_{\text{latency}}$
  • Workflow Quality Constraint: $\prod_{v \in \mathcal{V}} \left( \sum_{k \in \mathcal{K}v} x{v,k} \cdot A_{v,k} ight) \ge \text{SLO}_{\text{accuracy}}$
  • Energy Budget (Optional): $\sum_{v \in \mathcal{V}} \sum_{k \in \mathcal{K}v} x{v,k} \cdot E_{v,k} \le \text{Budget}_{\text{energy}}$

Here, $\mathcal{P}_{\text{crit}}$ denotes the critical path through the workflow DAG. Because typical agent workflows contain between 4 and 25 tasks, Murakkab’s solver resolves this ILP in under 5 milliseconds, generating an optimal static configuration plan.

The Adaptive Runtime: Dynamic Slack-Aware Reconfiguration

Static plans derived from offline profiling provide an excellent baseline, but production execution is inherently dynamic. External tools, network latency, and non-deterministic agent tool loops cause runtime execution times to deviate from profiled estimates.

Murakkab’s Adaptive Runtime solves this through real-time Dynamic Reconfiguration.

Real-Time Slack Monitoring

Let $T_v^{\text{actual}}$ denote the actual completion time of task $v$. As tasks complete along the DAG, Murakkab computes the Remaining Latency Slack $\mathcal{S}(t)$ for the remaining downstream path $\mathcal{P}_{\text{rem}}$:

$$ \mathcal{S}(t) = \text{SLO}{\text{latency}} - \left( t - t{\text{start}} ight) - \sum_{u \in \mathcal{P}{\text{rem}}} \hat{L}{u} $$

where $\hat{L}_{u}$ is the expected latency of pending task $u$ under the current execution plan.

+-------------------------------------------------------------------------------+ | MURAKKAB DYNAMIC SLACK RECONFIGURATION | +-------------------------------------------------------------------------------+ | | | Scenario 1: Upstream Tool Delayed (Slack Deficit) | | Task T1 (Classifier): Finished in 50ms (Normal) | | Task T2 (Search Tool): Experienced network lag! Took 650ms (Expected 150ms) | | Current Slack: NEGATIVE (-500ms deficit!) | | | | Dynamic Action: | | Murakkab reconfigures Task T3: Switches from 70B-FP16 to 70B-FP8 + FlashInfer | | Task T3 latency drops from 1200ms -> 600ms, recovering 600ms of slack! | | Result: Workflow completes in 2.1s (Under 2.5s SLO) -> Zero SLO Miss! | | | | Scenario 2: Upstream Tool Ultra-Fast (Slack Surplus) | | Task T2 finishes in 40ms. Slack surplus = +200ms. | | Dynamic Action: | | Murakkab downgrades Task T4 to a cheaper SLM on an L40S GPU. | | Result: Slashes cloud cost while still beating the 2.5s deadline. | +-------------------------------------------------------------------------------+

By continuously steering downstream tasks across the Pareto frontier, Murakkab absorbs external tool variance and guarantees high SLO attainment while driving infrastructure costs to the mathematical minimum.

Python Implementation: Building a Murakkab Workflow Orchestrator

To demonstrate how Murakkab’s declarative abstractions, Pareto optimization, and dynamic slack reconfiguration work in practice, let us construct a fully functional simulation in Python and PyTorch.

Our implementation includes:

  1. TaskVariant: Encapsulates model, quantization, hardware target, accuracy, latency, and cost metadata.
  2. DeclarativeWorkflow: Graph abstraction managing tasks, dependencies, and end-to-end SLOs.
  3. ProfileGuidedOptimizer: Solves the Pareto frontier across candidates to generate execution plans.
  4. AdaptiveRuntime: Executes the workflow DAG, monitors remaining slack, and reconfigures downstream tasks dynamically.
import time import random from typing import List, Dict, Optional, Tuple class TaskVariant: # An execution configuration candidate for a workflow task def __init__( self, name: str, model_name: str, precision: str, hardware: str, accuracy: float, latency_ms: float, cost_cents: float, ): self.name = name self.model_name = model_name self.precision = precision self.hardware = hardware self.accuracy = accuracy self.latency_ms = latency_ms self.cost_cents = cost_cents class WorkflowTask: # A single node in the agentic workflow DAG def __init__(self, task_id: str, description: str, depends_on: Optional[List[str]] = None): self.task_id = task_id self.description = description self.depends_on = depends_on or [] self.variants: List[TaskVariant] = [] self.selected_variant: Optional[TaskVariant] = None def add_variant(self, variant: TaskVariant): self.variants.append(variant) class ProfileGuidedOptimizer: # Solves the optimal execution plan across tasks to minimize cost under SLO def __init__(self, slo_latency_ms: float, min_accuracy: float): self.slo_latency_ms = slo_latency_ms self.min_accuracy = min_accuracy def optimize_plan(self, tasks: List[WorkflowTask]) -> Dict[str, TaskVariant]: # Evaluates candidate combinations to find minimum cost meeting constraints best_plan = {} best_cost = float('inf') # Simple greedy solver for demonstration def search(idx: int, current_plan: Dict[str, TaskVariant], total_lat: float, total_acc: float, total_cost: float): nonlocal best_cost, best_plan if idx == len(tasks): if total_lat <= self.slo_latency_ms and total_acc >= self.min_accuracy: if total_cost < best_cost: best_cost = total_cost best_plan = dict(current_plan) return task = tasks[idx] for v in task.variants: current_plan[task.task_id] = v search( idx + 1, current_plan, total_lat + v.latency_ms, total_acc * v.accuracy, total_cost + v.cost_cents ) search(0, {}, 0.0, 1.0, 0.0) return best_plan class AdaptiveRuntime: # Executes the workflow DAG and dynamically reconfigures tasks based on slack def __init__(self, tasks: List[WorkflowTask], slo_latency_ms: float, pgo: ProfileGuidedOptimizer): self.tasks = tasks self.slo_latency_ms = slo_latency_ms self.pgo = pgo self.task_map = {t.task_id: t for t in tasks} def execute_workflow(self, simulate_tool_delay_ms: float = 0.0) -> Dict[str, float]: start_time = time.time() elapsed_sim_ms = 0.0 total_cost = 0.0 reconfigurations = 0 # Initial static optimization current_plan = self.pgo.optimize_plan(self.tasks) for t in self.tasks: t.selected_variant = current_plan.get(t.task_id, t.variants[0]) for idx, task in enumerate(self.tasks): # Calculate remaining slack before executing task remaining_expected_ms = sum( self.task_map[rem_t.task_id].selected_variant.latency_ms for rem_t in self.tasks[idx:] ) slack_ms = self.slo_latency_ms - (elapsed_sim_ms + remaining_expected_ms) # Dynamic Reconfiguration: If in negative slack, upgrade to faster variant if slack_ms < -50.0: # Find faster variant for current task faster_variants = sorted(task.variants, key=lambda v: v.latency_ms) if faster_variants[0].latency_ms < task.selected_variant.latency_ms: task.selected_variant = faster_variants[0] reconfigurations += 1 # Simulate execution of task executed_variant = task.selected_variant actual_task_latency = executed_variant.latency_ms # Inject external tool delay if this task is a retrieval tool if "tool" in task.task_id: actual_task_latency += simulate_tool_delay_ms elapsed_sim_ms += actual_task_latency total_cost += executed_variant.cost_cents is_slo_met = elapsed_sim_ms <= self.slo_latency_ms return { "total_latency_ms": elapsed_sim_ms, "total_cost_cents": total_cost, "slo_met": is_slo_met, "reconfigurations": reconfigurations }

Architectural Comparison: Murakkab vs. Traditional Workflow Frameworks

To contextualize Murakkab’s systems achievements, the table below evaluates Murakkab against prevailing client-side agent frameworks and standard LLM serving solutions across critical enterprise dimensions:

Architectural DimensionClient-Side Frameworks (LangChain/CrewAI)Multi-Model Gateways (LiteLLM/Envoy)Standard Cloud Schedulers (Kubernetes)Murakkab (OSDI 2026)
Workflow DAG VisibilityClient-only (Opaque to Cloud)Request-level onlyPod-level onlyFull End-to-End Cloud DAG
Model SpecificationHardcoded per nodePer-route fallbackNot model-awareDeclarative Intent & Constraints
Pareto OptimizationNone (Ad-hoc manual choice)Simple latency or cost routingStatic container sizingMulti-Objective PGO Solver
Real-Time Slack RebalancingNone (Static execution)None (Isolated calls)None (Pod autoscale only)Dynamic Runtime Reconfiguration
Hardware MappingFixed per API endpointFixed per backend endpointStatic GPU node assignmentHeterogeneous Stage Matching
Tool Jitter ResilienceZero (Causes deadline misses)Zero (Blind to dependencies)ZeroAbsorbs tool delay online
Cloud Cost ReductionBaseline (1.0x)1.1x–1.3x1.0xUp to 4.3x Cost Reduction

While multi-model routing proxies (like LiteLLM or RouteLLM) can direct single queries to cheaper models, they lack structural awareness of multi-step dependencies. Murakkab uniquely unifies DAG dependency analysis with full-stack model and hardware steering.

Empirical Benchmark Results & Production Validation

In the USENIX OSDI 2026 proceedings, Chaudhry et al. evaluated Murakkab across extensive multi-agent production workloads drawn from Microsoft Azure telemetry, spanning video question-answering, iterative code synthesis, multi-hop RAG, and mathematical problem-solving:

1. Cloud Cost and Energy Reduction

Compared against standard production baseline deployments where workflows are served using statically configured models on dedicated GPU clusters:

Cloud Serving Cost per 1,000 Agentic Workflows ($ USD) Workload: Enterprise Video QA & Code Synthesis Workflows Static Production Baseline (GPT-4o / 70B): |==============================> $18.40 (1.00x) Naive Model Sizing (Ad-Hoc Manual): |======================> $12.10 (1.52x) Murakkab (OSDI 2026): |=======> $4.28 (4.30x Reduction!)

Murakkab slashed operational cloud serving expenditure by 4.3x while decreasing datacenter energy consumption by 3.7x, unlocking multi-million-dollar annual savings for hyperscale cloud deployments.

2. GPU Hardware Footprint Compression

By mapping non-critical intermediate steps (e.g., classification, extraction, validation) to smaller quantized SLMs hosted on cost-effective L40S accelerators and CPUs, Murakkab reduced total cluster GPU demand by 2.8x. Cloud providers can host nearly triple the volume of concurrent agentic applications on existing datacenter infrastructure.

3. End-to-End SLO Compliance Under Tool Jitter

In real-world environments, external tool dependencies (database queries, code execution sandboxes, live web crawling) induce heavy tail latency spikes:

  • Under static execution frameworks, tool latency spikes caused the end-to-end workflow SLO violation rate to exceed $44.8%$.
  • Under Murakkab’s adaptive runtime, dynamic slack rebalancing dynamically adjusted downstream model variants, maintaining an SLO compliance rate of $99.4%$.

Step-by-Step Implementation Guide: Adopting Murakkab Principles

For AI systems architects, machine learning engineers, and cloud infrastructure platform teams seeking to adopt Murakkab’s design patterns, follow this phased deployment blueprint:

Step 1: Transition from Hardcoded Scripts to Declarative DAGs

Refactor your agent application code to decouple business logic from concrete model checkpoints. Express workflow nodes as semantic tasks with explicit latency and accuracy bounds:

{ "workflow_id": "customer_support_pipeline", "slo": { "latency_max_ms": 3000, "quality_floor": 0.9 }, "nodes": [ { "id": "intent_parser", "type": "classification", "candidates": ["llama-3-8b-int4", "mistral-7b-fp8", "llama-3-70b-fp16"] }, { "id": "solution_generator", "type": "reasoning", "depends_on": ["intent_parser"], "candidates": ["qwen-2.5-72b-fp8", "deepseek-v3-fp8", "llama-3-70b-fp16"] } ] }

Step 2: Establish Offline Profiling Pipelines

Profile candidate model variants and hardware configurations on representative prompt workloads. Record empirical distributions for:

  • Prefill time per token
  • Decode time per token
  • Output accuracy / benchmark F1 score
  • Cost per 1,000 tokens

Store these profiled parameters in your centralized orchestration metadata database.

Step 3: Implement Upstream Delay Traps in the Execution Gateway

In your central workflow runner, measure the elapsed time of external tool calls. If an external tool experiences unexpected latency, recalculate the critical path slack and dynamically route the subsequent node to an accelerated model endpoint:

# Upstream delay detection & dynamic route adjustment def dispatch_next_stage(workflow_context, current_node): elapsed_ms = workflow_context.get_elapsed_time_ms() remaining_budget_ms = workflow_context.slo_latency_ms - elapsed_ms if remaining_budget_ms < current_node.default_expected_latency_ms: # Emergency acceleration: select fastest candidate selected_candidate = current_node.get_fastest_candidate() else: # Standard cost-optimal selection selected_candidate = current_node.get_cheapest_feasible_candidate(remaining_budget_ms) return invoke_model_endpoint(selected_candidate, workflow_context.payload)

For enterprises building automated high-throughput scraping pipelines using our data fetching infrastructure or enterprise agent copilots, applying these workflow-aware orchestration principles yields massive reductions in operational cloud expenditures. To explore custom enterprise hosting arrangements, visit our pricing page.

Future Outlook: The Evolution of Cloud-Native Agent Operating Systems

The publication of Murakkab at USENIX OSDI 2026 marks the arrival of Cloud-Native Agent Operating Systems. Just as relational database management systems (RDBMS) decoupled SQL queries from physical disk B-tree layouts in the 1970s, modern agent serving must decouple semantic workflow graphs from physical GPU kernel executions.

Looking toward 2027 and beyond, we anticipate several vital architectural evolutions:

  1. Cross-Workflow Batching & KV Cache Sharing: Murakkab expanding to merge identical sub-DAG paths across unrelated users, enabling cross-workflow prefix caching and multi-tenant batching.
  2. Carbon-Aware Agentic Scheduling: Automatically migrating non-urgent agentic workflow tasks to datacenters with surplus renewable energy while routing latency-critical steps to nearby low-latency edge accelerators.
  3. Hardware-Synthesized Tool Accelerators: Datacenters co-locating lightweight WebAssembly (WASM) microVM runtimes directly beside GPU clusters to reduce tool invocation latency to sub-millisecond timescales.

For further technical reading, systems architectural guides, and engineering blueprints, check out the documentation hub or discover how AIFetchly accelerates AI engineering teams across the globe.

Conclusion & Strategic Takeaways

As Compound AI Systems become the dominant paradigm for enterprise intelligence, treating agentic workflows as opaque client-side scripts is an unsustainable operational antipattern.

The strategic takeaways from Murakkab include:

  • Client-Side Frameworks Hide the Graph: Cloud schedulers cannot optimize what they cannot see. Exposing agentic workflows as declarative DAGs unlocks full-stack cross-layer optimization.
  • Decouple Specification from Execution: Never hardcode models or hardware in workflow nodes. Let profile-guided optimizers navigate the accuracy-latency-cost Pareto frontier.
  • Dynamic Slack Monitoring Eliminates Stragglers: Upstream tool delays no longer cause SLO violations when downstream tasks can dynamically reconfigure in real time.
  • Massive Cloud ROI: Delivering a 4.3x reduction in cloud serving costs and a 2.8x reduction in GPU footprint, Murakkab establishes the new standard for resource-efficient agentic infrastructure in 2026.