mixture-of-agentsmoamulti-agenttogether-aillm-collaborationtutorial

Mixture-of-Agents: Scaling LLM Quality with Layered MoA

September 23, 2026

Mixture-of-Agents (MoA) Architecture: Layered Feed-Forward Proposers, Cross-Model Synthesis, and Master Aggregators

The Problem: The Single-Model Plateau and Scaling Bottlenecks

In the pursuit of higher reasoning fidelity, code generation accuracy, and automated research capabilities in platforms like AiFetchly, the AI industry has historically relied on pre-training scaling laws: training ever-larger monolithic models on tens of trillions of tokens.

However, as foundation models push beyond hundreds of billions of parameters, pre-training scaling encounters severe diminishing returns:

Exponential Compute Requirements: Achieving marginal reductions in evaluation loss requires orders of magnitude more GPU cluster hours and megawatts of power.
Knowledge Specialization Gaps: No single foundation model dominates across all domains. One model may excel at mathematical reasoning (e.g., Qwen-2), another at Python software engineering (e.g., LLaMA-3), and a third at structured data extraction or creative prose (e.g., Mixtral).
The Brittleness of Standalone Generation: When a single model generates an answer, any hallucination or logical misstep introduced early in the autoregressive stream cascades into the final output.

Limitations of Conventional Ensembling and Best-of-N:
┌─────────────────────────┬─────────────────────────────────────────────────────────┐
│ Strategy                │ Structural Limitation                                   │
├─────────────────────────┼─────────────────────────────────────────────────────────┤
│ Best-of-N Sampling      │ Discards N-1 outputs! Wastes 90% of generated tokens.   │
│ Majority Voting         │ Only works for closed-ended answers (e.g., math/logic). │
│ Multi-Agent Debate      │ Unbounded conversational rounds; high latency and loops.│
│ Monolithic Scaling      │ Prohibitive training costs; centralized vendor lock-in. │
└─────────────────────────┴─────────────────────────────────────────────────────────┘

1. Why Best-of-N and Majority Voting Fall Short

To improve generation quality at test-time, developers frequently turn to Best-of-N sampling or majority voting:

In Best-of-N, the system samples $N$ candidate responses from a model, scores each candidate using a reward model or automated judge, and selects the highest-scoring candidate.
While effective, Best-of-N is fundamentally wasteful: if Candidate 1 has an exceptional introduction, Candidate 2 provides the correct algorithm implementation, and Candidate 3 identifies a critical security vulnerability, Best-of-N selects one candidate and discards the valuable insights contained in the remaining $N-1$ responses.
In Majority Voting, models vote on categorical or numeric answers (e.g., selecting the most frequent answer in GSM8K math). However, majority voting cannot synthesize open-ended technical architecture proposals, analytical essays, or complex software refactoring tasks.

2. The Unexplored Boundary: Collective Multi-Model Intelligence

If human engineering teams solve complex architectural problems by having diverse specialists draft proposals, review peer ideas, and synthesize a cohesive plan, can heterogeneous language models collaborate in a structured, feed-forward topology?

Until recently, multi-agent frameworks suffered from conversational drifting, high latency, and unstructured consensus loops. A principled architectural framework was needed to systematically harness the collective intelligence of multiple models.

The Architecture: Mixture-of-Agents (MoA)

To overcome the single-model performance plateau, researchers from Together AI and Duke University introduced Mixture-of-Agents (MoA) (Wang et al., 2024).

Mixture-of-Agents organizes multiple language models into a layered, feed-forward Directed Acyclic Graph (DAG) where agents alternate between acting as proposers and aggregators.

Mixture-of-Agents Layered Workflow: Parallel Proposers, Intermediate Refiners, and Master Aggregator Adjudication

1. The Collaborativeness Phenomenon

The empirical discovery underpinning MoA is what the authors termed the Collaborativeness Phenomenon:

Large Language Models generate significantly higher-quality responses when presented with outputs from other models, even when the auxiliary models are strictly inferior or smaller than the evaluating model.

When an LLM is prompted with candidate answers from diverse peer models:

It acts as an effective critic and synthesizer, identifying factual discrepancies, reconciling conflicting approaches, and extracting the most robust arguments from each source.
Even weak models provide alternative vantage points, edge case considerations, or vocabulary cues that trigger deeper reasoning paths in stronger aggregator models.

2. The Layered Feed-Forward DAG Topology

Unlike circular conversational debate frameworks that iterate unpredictably, MoA enforces a deterministic, layered architecture structured as follows:

Let $q$ be the user query. The MoA architecture consists of $L$ layers:

$$\text{Layer } l \in {1, 2, \dots, L}$$

Each layer $l$ comprises $N_l$ distinct LLM agents:

$$\mathcal{M}^{(l)} = {M_1^{(l)}, M_2^{(l)}, \dots, M_{N_l}^{(l)}}$$

Layer 1: Independent Proposers: In the initial layer, each agent $M_i^{(1)}$ independently receives the original user query $q$ and generates an initial candidate response in parallel:

$$y_i^{(1)} = M_i^{(1)}(q), \quad \forall i \in {1, \dots, N_1}$$

Because these models run concurrently, the latency of Layer 1 is simply the maximum execution time among the $N_1$ models:

$$T_{\text{Layer 1}} = \max_i \left( T(M_i^{(1)}) \right)$$

Layers 2 to L-1: Intermediate Refiners: In subsequent intermediate layers, each agent $M_i^{(l)}$ receives both the original user query $q$ and the full set of candidate responses generated by all models in the preceding layer:

$$\mathcal{Y}^{(l-1)} = {y_1^{(l-1)}, y_2^{(l-1)}, \dots, y_{N_{l-1}}^{(l-1)}}$$

The prompt provided to agent $M_i^{(l)}$ is structured as:

$$\text{Prompt}_i^{(l)} = \text{Format}(q, \mathcal{Y}^{(l-1)})$$

$$y_i^{(l)} = M_i^{(l)}(\text{Prompt}_i^{(l)}), \quad \forall i \in {1, \dots, N_l}$$

Each agent in layer $l$ compares, critiques, and refines the candidate outputs, producing an updated response that incorporates the best elements of the previous layer.

Layer L: The Master Aggregator: In the final layer, a designated Master Aggregator LLM (typically the strongest available model, such as Qwen-2 72B, LLaMA-3 70B, or GPT-4o) evaluates all outputs from Layer $L-1$ and produces the authoritative, unified final response:

$$y^* = M_{\text{Aggregator}}^{(L)}\left(\text{Format}(q, \mathcal{Y}^{(L-1)})\right)$$

3. Prompt Engineering for Synthesis and Critique

The prompt template used to format peer responses for intermediate refiners and the final aggregator is critical to avoiding superficial regurgitation:

You have been provided with a set of responses from other models to the original user query.
Your task is to act as an expert synthesizer and editor:
1. Identify any factual inaccuracies, logical fallacies, or edge cases missed by individual models.
2. Combine the strongest arguments, code snippets, and structural insights from all responses.
3. Formulate a comprehensive, definitive, and well-structured final response.

[Original Query]
{user_query}

[Responses from Other Models]
Response 1: {response_1}
Response 2: {response_2}
Response 3: {response_3}
Response 4: {response_4}

[Final Synthesized Output]

4. Heterogeneous vs. Homogeneous MoA Configurations

MoA supports two distinct operational modes:

Heterogeneous MoA: Combines models from different families and architectures (e.g., Qwen-2 72B, LLaMA-3 70B, Mixtral 8x22B, and Gemma-2 27B). Heterogeneous MoA achieves the highest performance gains because different training datasets, tokenizers, and alignment recipes maximize cognitive diversity.
Homogeneous MoA: Uses multiple instances of the same base model (e.g., 4 instances of LLaMA-3 70B sampled at temperature $T=0.7$). Even without architectural diversity, sampling variance generates distinct solution paths that the subsequent aggregation layer successfully merges into a superior response.

Comparative Breakdown: Multi-Model Architectures

Dimension	Mixture-of-Experts (MoE)	Best-of-N Sampling	Multi-Agent Debate (MAD)	Mixture-of-Agents (MoA)
Operational Level	Internal Layer (FFN routing)	Post-Generation Selection	Conversational Chat Loop	Layered Feed-Forward DAG
Model Scope	Single Model Weights	Single Model Rollouts	Multi-Model Discourse	Heterogeneous Model Black-Boxes
Synthesis Mechanism	Softmax-Weighted Activations	None (Rejects $N-1$ outputs)	Multi-Round Adjudication	Multi-Layer Context Synthesis
Execution Latency	Near-Zero Overhead	$O(N)$ Sequential or Parallel	High (Unbounded loops)	Bounded: $L \times \max(T_{\text{agent}})$
AlpacaEval 2.0 Win Rate	Baseline	+2% to +4%	+3% to +5%	65.1% (+7.6% over GPT-4o)

Implementation: Building an Asynchronous Mixture-of-Agents Engine in Python

Let us construct a clean, modular reference implementation of the Mixture-of-Agents architecture in Python using asyncio.

We will implement:

The MoAAgent Interface: Represents an asynchronous language model client capable of zero-shot generation and peer response synthesis.
The MixtureOfAgents Engine: Manages multi-layer execution, constructs synthesis prompts, and coordinates concurrent model calls per layer.
Verification & Quality Benchmark: Simulates multi-layer execution on a complex technical prompt, demonstrating how collaborative synthesis progressively elevates answer quality.

import asyncio
import numpy as np
from typing import List, Dict, Any, Optional

class MoAAgent:
    """
    Represents an LLM agent within the Mixture-of-Agents topology.
    Capable of zero-shot proposal generation and multi-candidate synthesis.
    """
    def __init__(self, name: str, base_capability: float, latency_sec: float = 0.5):
        self.name = name
        self.base_capability = base_capability
        self.latency_sec = latency_sec

    def build_synthesis_prompt(self, user_query: str, peer_candidates: List[Dict[str, Any]]) -> str:
        """Constructs the prompt formatting peer candidate responses for synthesis."""
        prompt = f"Original Query: {user_query}\n\n"
        prompt += "Below are candidate responses from peer models. Critique, synthesize, "
        prompt += "and produce a superior, hardened final solution:\n\n"
        for idx, cand in enumerate(peer_candidates, 1):
            prompt += f"--- Candidate {idx} ({cand['model']}) ---\n{cand['content']}\n\n"
        return prompt

    async def generate(
        self,
        user_query: str,
        peer_candidates: Optional[List[Dict[str, Any]]] = None,
    ) -> Dict[str, Any]:
        """Simulates asynchronous LLM inference with collaborativeness synergy."""
        await asyncio.sleep(self.latency_sec * 0.05)  # Simulated API network latency

        if not peer_candidates:
            # Layer 1: Zero-shot proposal generation
            score = self.base_capability + np.random.normal(0.0, 0.01)
            content = f"[{self.name}] Initial technical proposal addressing: '{user_query[:35]}...'"
        else:
            # Layer > 1: The Collaborativeness Phenomenon
            # Model synthesizes best ideas from peers, achieving a score superior to prior mean
            peer_scores = [c["score"] for c in peer_candidates]
            best_peer_score = max(peer_scores)
            mean_peer_score = np.mean(peer_scores)

            # Mathematical synergy boost: combines peer strengths + self reasoning
            synergy = 0.04 * (1.0 - best_peer_score) + 0.02 * (best_peer_score - mean_peer_score)
            score = best_peer_score + synergy + np.random.normal(0.0, 0.005)
            content = f"[{self.name}] Synthesized solution integrating insights from {len(peer_candidates)} peer models."

        return {
            "model": self.name,
            "content": content,
            "score": float(min(0.98, max(0.1, score))),
        }


class MixtureOfAgentsEngine:
    """
    Coordinates execution of an L-layer Mixture-of-Agents Directed Acyclic Graph (DAG).
    """
    def __init__(self, proposer_layers: List[List[MoAAgent]], aggregator_agent: MoAAgent):
        self.proposer_layers = proposer_layers
        self.aggregator = aggregator_agent

    async def execute(self, user_query: str) -> Dict[str, Any]:
        """Executes the complete layered MoA pipeline concurrently."""
        current_candidates: List[Dict[str, Any]] = []
        execution_history = []

        # Execute intermediate proposer layers
        for layer_idx, layer_agents in enumerate(self.proposer_layers, 1):
            tasks = [
                agent.generate(
                    user_query=user_query,
                    peer_candidates=current_candidates if layer_idx > 1 else None,
                )
                for agent in layer_agents
            ]
            # Execute all agents within the layer in parallel
            current_candidates = await asyncio.gather(*tasks)

            mean_s = np.mean([c["score"] for c in current_candidates])
            max_s = np.max([c["score"] for c in current_candidates])
            execution_history.append({
                "layer": layer_idx,
                "num_agents": len(layer_agents),
                "mean_score": mean_s,
                "best_score": max_s,
            })
            print(f"Layer {layer_idx} ({len(layer_agents)} Models Parallel): Mean Score = {mean_s:.3f} | Best Score = {max_s:.3f}")

        # Execute Master Aggregator Layer
        final_output = await self.aggregator.generate(
            user_query=user_query,
            peer_candidates=current_candidates,
        )
        print(f"Final Layer (Master Aggregator: {self.aggregator.name}): Authority Score = {final_output['score']:.3f}")

        return {
            "final_response": final_output["content"],
            "final_score": final_output["score"],
            "history": execution_history,
        }


async def run_moa_verification():
    """Verifies the Mixture-of-Agents architecture and measures quality gain."""
    print("=" * 70)
    print("MIXTURE-OF-AGENTS (MoA) LAYERED SYNTHESIS BENCHMARK")
    print("=" * 70)

    # Configure a 3-layer MoA topology
    # Layer 1: 4 diverse open-weight proposers
    layer1_proposers = [
        MoAAgent("Qwen-2-72B", base_capability=0.73, latency_sec=0.7),
        MoAAgent("LLaMA-3-70B", base_capability=0.75, latency_sec=0.8),
        MoAAgent("Mixtral-8x22B", base_capability=0.69, latency_sec=0.6),
        MoAAgent("Gemma-2-27B", base_capability=0.66, latency_sec=0.5),
    ]

    # Layer 2: 3 intermediate refiners
    layer2_refiners = [
        MoAAgent("Qwen-2-72B", base_capability=0.73, latency_sec=0.7),
        MoAAgent("LLaMA-3-70B", base_capability=0.75, latency_sec=0.8),
        MoAAgent("Mixtral-8x22B", base_capability=0.69, latency_sec=0.6),
    ]

    # Layer 3: Master Aggregator
    master_aggregator = MoAAgent("Qwen-2-72B-Instruct", base_capability=0.76, latency_sec=1.0)

    moa_engine = MixtureOfAgentsEngine(
        proposer_layers=[layer1_proposers, layer2_refiners],
        aggregator_agent=master_aggregator,
    )

    sample_prompt = "Architect a resilient, distributed event-sourcing data pipeline using Kafka and Rust."
    result = await moa_engine.execute(sample_prompt)

    best_solo_model = max(agent.base_capability for agent in layer1_proposers)
    quality_improvement = ((result["final_score"] - best_solo_model) / best_solo_model) * 100.0

    print("\n" + "=" * 70)
    print("BENCHMARK RESULTS & QUALITY MULTIPLIER")
    print("=" * 70)
    print(f"Best Standalone Individual Model:  {best_solo_model:.3f}")
    print(f"Final MoA Synthesized Score:      {result['final_score']:.3f}")
    print(f"Quality Improvement Percentage:   +{quality_improvement:.2f}% over best individual model!")
    print("=" * 70)
    assert result["final_score"] > best_solo_model, "MoA must outperform the best standalone model"
    print("  [SUCCESS] Mixture-of-Agents collaborativeness verified successfully!")


if __name__ == "__main__":
    asyncio.run(run_moa_verification())

Production Advantages and Engineering Considerations

Deploying Mixture-of-Agents in mission-critical applications introduces key strategic and operational advantages:

1. Beating Closed Frontier Models with Open-Weight Sovereignty

The most compelling achievement of Together AI’s MoA is proving that an ensemble of open-weight models can surpass state-of-the-art proprietary systems:

On AlpacaEval 2.0, an MoA built entirely with open-source foundation models achieved a 65.1% win rate, soundly defeating standalone GPT-4o (57.5%).
For regulated industries (healthcare, finance, defense) that require strict data sovereignty, MoA allows organizations to deploy on-premise open-source models while achieving frontier intelligence.

2. Radical Reduction in Hallucination and Bias

Because each layer cross-references independent responses:

Isolated hallucinations introduced by one model are identified and filtered out by peer refiners in subsequent layers.
Factual grounding and instruction adherence increase substantially because the master aggregator penalizes claims that lack cross-model consensus.

3. Latency Optimization via Sparse MoA (SMoA)

While MoA requires multiple model calls, total end-to-end latency is bounded because all models within a layer execute concurrently:

Total latency scales with the number of layers ($L$), not the total number of agents ($N \times L$).
Advanced variants like Sparse Mixture-of-Agents (SMoA) further optimize cost by scoring Layer 1 candidates with a lightweight reward filter, forwarding only the top-$k$ most informative responses to subsequent layers, cutting token consumption by 40% to 50%.

Conclusion

Mixture-of-Agents (MoA) shifts the AI scaling paradigm from monolithic model parameter expansion to collaborative multi-model orchestration. By formalizing the Collaborativeness Phenomenon into a structured, feed-forward DAG of parallel proposers and sequential aggregators, MoA synthesizes the complementary strengths of diverse foundation models.

Delivering state-of-the-art reasoning quality that outperforms standalone proprietary models, MoA provides AI systems architects and enterprise developers with a robust, scalable blueprint for building reliable, high-intelligence agentic systems.