arize-phoenixllm-observabilityopentelemetryrag-evaluationai-agentsevalstutorial

Arize Phoenix Guide: OpenTelemetry Tracing & LLM Evals

Arize Phoenix Architecture: OpenTelemetry Distributed Tracing, LLM-as-a-Judge Automated Evals, UMAP Embedding Visualizations, and Zero-Exfiltration Self-Hosting

The Problem: The “Black Box” Failure Mode in Production LLMs and Agents

In modern enterprise generative artificial intelligence, multi-agent frameworks, and autonomous web harvesting tools powered by AiFetchly, applications have moved far beyond simple single-turn prompt-response scripts. Production systems are complex, multi-stage computational graphs: ingesting web pages, parsing PDF layouts, routing queries through vector databases, invoking third-party APIs via tool-calling loops, and synthesizing structured outputs.

However, operationalizing these non-deterministic agentic architectures reveals a glaring systems engineering deficit: the complete failure of traditional Application Performance Monitoring (APM) tools.

The GenAI "Silent Failure" Dilemma: ┌─────────────────────────────────────────────────────────────────┐ │ User Query: "Extract Q3 operating income from uploaded 10-K" │ └─────────────────────────────────────────────────────────────────┘ │ ▼ [Multi-Step Agentic RAG Pipeline] ├── Step 1: Web / PDF Fetching ──► HTTP 200 OK (50ms) ├── Step 2: Vector Search ──► HTTP 200 OK (15ms, Cosine: 0.82) ├── Step 3: Tool Execution ──► Exit Code 0 (120ms) └── Step 4: Foundation LLM Call ──► HTTP 200 OK (1,200ms, 850 Tokens) │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Traditional APM Verdict (Datadog, Prometheus, New Relic): │ │ ✅ "All Endpoints Healthy: 100% Uptime, Low Latency, HTTP 200s" │ ├─────────────────────────────────────────────────────────────────┤ │ The Ground Truth Reality: │ │ ❌ Step 2 retrieved an outdated 2023 financial table! │ │ ❌ Step 3 extracted gross profit instead of operating income! │ │ ❌ Step 4 hallucinated a confident $1.2B revenue projection! │ │ ❌ User receives completely false data with zero error logs! │ └─────────────────────────────────────────────────────────────────┘

1. The Blind Spots of Traditional Infrastructure APM

Standard observability systems (Datadog, Dynatrace, Prometheus) were designed around deterministic microservices:

  • They measure CPU load, RAM allocation, network I/O, database query latency, and HTTP response status codes.
  • The Semantic Blind Spot: In an LLM application, an endpoint can return an HTTP 200 OK in 800 milliseconds while delivering a completely fabricated hallucination, leaking private API keys, or generating toxic content.
  • Traditional APMs cannot inspect prompt token structures, evaluate grounding against retrieved context chunks, or diagnose why an autonomous agent entered an infinite tool-calling loop.

2. The Data Governance and Privacy Risks of SaaS Observability

To gain AI visibility, many software engineering teams turned to proprietary cloud observability platforms:

  • Data Exfiltration: Every prompt, system instruction, customer input, retrieved database snippet, and LLM completion must be streamed across the public internet to a third-party SaaS server.
  • In privacy-sensitive enterprise pipelines—such as handling proprietary codebases, confidential customer legal contracts, or healthcare diagnostics—uploading unredacted context violates data sovereignty laws (GDPR, HIPAA, SOC 2).
  • Vendor Lock-In: Proprietary platforms enforce closed, vendor-specific SDK wrappers that make migrating between cloud providers or evaluation frameworks prohibitively expensive.

3. The Lack of Scientific Evaluation in CI/CD Deployments

When developers update an LLM prompt, adjust a chunking strategy in a vector database, or swap foundation model weights (e.g., migrating from GPT-4o to LLaMA-3.3 70B):

  • Traditional software uses unit tests with exact assertions (assert output == expected).
  • Because natural language output is stochastic, exact string matching fails. Without automated, statistical LLM-as-a-Judge evaluations, engineering teams resort to “vibe checks,” deploying prompt modifications that silently break edge cases and cause widespread retrieval regressions.

The Solution: Arize Phoenix’s OpenTelemetry-Native Architecture

To bring open, standards-based observability and rigorous evaluation to generative AI, Arize AI created Phoenix.

Arize Phoenix is an open-source, vendor-neutral AI observability and evaluation platform engineered specifically for LLMs, agentic workflows, and RAG pipelines. Built entirely on OpenTelemetry (OTel) GenAI semantic conventions, Phoenix provides distributed tracing across every token generation, tool execution, and vector retrieval, coupled with automated LLM-as-a-Judge evaluations and UMAP latent embedding visualizations.

Arize Phoenix Architecture: ┌─────────────────────────────────────────────────────────────────┐ │ AI Applications (aiFetchly Desktop, Web Agents, Multi-Agentic) │ └─────────────────────────────────────────────────────────────────┘ │ ▼ (OTel Standard Auto-Instrumentation: arize-phoenix-otel) ┌─────────────────────────────────────────────────────────────────┐ │ OpenTelemetry GenAI Semantic Collector │ │ • Captures Traces: Root Spans, LLM Invocations, Vector Search │ │ • Records Metadata: Token counts, TTFT, temperature, costs │ │ • Tracks Tool Calling: Input arguments, outputs, exceptions │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Phoenix Storage & Analysis Core (100% Self-Hosted & Local) │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Hierarchical Distributed Trace Visualizer │ │ │ │ • Waterfall latency graphs across nested agent workflows │ │ │ │ • Token usage & financial cost heatmaps │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ │ │ │ ┌───────────────────────┴───────────────────────┐ │ │ ▼ ▼ │ │ ┌─────────────────────────────┐ ┌─────────────────────────────┐ │ │ │ Automated Evaluation Suite │ │ UMAP Embedding Explorer │ │ │ │ • QA Correctness │ │ • 2D/3D Latent Clustering │ │ │ │ • Hallucination & Grounding │ │ • "The Silent Failure Map" │ │ │ │ • Context Relevance / Noise │ │ • Vector Retrieval Gaps │ │ │ │ • Toxicity / Safety Guards │ │ • Semantic Drift Detection │ │ │ └─────────────────────────────┘ └─────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Local Interactive UI / Jupyter / Docker / Enterprise Clusters │ └─────────────────────────────────────────────────────────────────┘

The Architectural Pillars of Arize Phoenix

  1. OpenTelemetry-Native by Design: Instead of relying on a proprietary SDK, Phoenix builds directly on the vendor-neutral OpenTelemetry standard (openinference / opentelemetry). Any telemetry emitted by Phoenix can be routed to standard OTel collectors, Grafana, Jaeger, or enterprise data warehouses.
  2. Hierarchical Trace Spans: Every interaction is decomposed into structured, parent-child span trees:
    • CHAIN: High-level workflow orchestration or agent goal.
    • LLM: Model invocation, token counts, temperature, raw prompts, completions.
    • RETRIEVER: Vector similarity search, candidate chunks, distance scores.
    • TOOL: External API calls, bash command execution, code sandbox runs.
  3. Automated LLM-as-a-Judge Evals: Provides pre-built and custom zero-shot evaluation templates that use frontier models to programmatically score output correctness, hallucinations, and context faithfulness.
  4. UMAP Latent Embedding Visualizations (“The Silent Failure Map”): Projects high-dimensional query and document vectors into interactive 2D and 3D geometric clusters using Uniform Manifold Approximation and Projection (UMAP), pinpointing semantic blind spots where vector databases consistently fail.
  5. 100% Open-Source & Zero Data Exfiltration: Runs locally via a single Python line (px.launch()) or a lightweight Docker container. All user prompts, sensitive customer data, and system traces remain strictly on local infrastructure.

Comparing Modern AI Observability Platforms

The table below contrasts Arize Phoenix against other leading observability engines:

AI Observability & Tracing Architectural Comparison: ┌───────────────────────────┬───────────────────────────────┬───────────────────────────────┬───────────────────────────────┐ │ Dimension │ Datadog / New Relic │ LangSmith │ Arize Phoenix │ ├───────────────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┤ │ Core Focus │ Infrastructure & APM microserv│ LangChain / LangGraph apps │ Open-source ML & GenAI obs │ │ Underlying Telemetry │ Proprietary APM agent │ Proprietary LangChain protocol│ Native OpenTelemetry (OTel) │ │ Deployment Model │ Cloud SaaS only │ Cloud SaaS (Self-host complex)│ 100% Self-Hosted (Python/Dock)│ │ Data Privacy & Sovereignty│ Transmits data to cloud │ Transmits data to cloud │ Zero data exfiltration │ │ Framework Agnostic │ Yes (HTTP only) │ Optimized for LangChain │ Completely agnostic (OTel) │ │ Embedding Visualization │ None │ Basic tabular views │ Native UMAP 2D/3D Failure Maps│ │ Automated Evals │ None (Threshold alerts only) │ LangChain evaluator templates │ Multi-model LLM-as-a-Judge │ │ Licensing │ Commercial subscription │ Proprietary commercial │ Open Source (Apache 2.0) │ └───────────────────────────┴───────────────────────────────┴───────────────────────────────┴───────────────────────────────┘

Deep Dive into Phoenix Core Primitives & Engines

Arize Phoenix delivers deep visibility into non-deterministic applications through four core subsystems: the OpenTelemetry Tracing Engine, Evaluation Protocols, UMAP Latent Space Analysis, and Continuous Testing Pipelines.

Phoenix Core Subsystem Architecture: Arize Phoenix Engine ├──► 1. OTel Tracing: Standardized hierarchical span trees for LLM, RAG, and Tools ├──► 2. LLM-as-a-Judge: Automated evaluation of Groundedness, QA, and Relevance ├──► 3. UMAP Failure Maps: 2D/3D dimensionality reduction revealing retrieval voids ├──► 4. Live Guardrails: Real-time toxicity, prompt injection, and drift detection └──► 5. Zero-Exfiltration Server: In-process SQLite / DuckDB storage with local UI

1. OpenTelemetry GenAI Semantic Conventions

Phoenix adopts the industry-standard OpenInference semantic conventions:

  • Trace Identity: Every user request generates a unique trace_id.
  • Span Categorization:
    • llm: Captures model name (gpt-4o, claude-3-5-sonnet, llama-3.3), input prompt messages, output completions, token counts (prompt, completion, total), and Time-to-First-Token (TTFT).
    • retriever: Captures input query text, retrieved document chunk IDs, raw text content, and vector similarity distance scores.
    • tool: Captures tool function names, JSON arguments, return payloads, and execution runtimes.
  • Trace Waterfall: Allows developers to inspect the exact critical path of an agent execution, immediately identifying which database call or LLM reflection loop caused a 5-second latency spike.

2. Automated LLM-as-a-Judge Evaluations

Rather than manually reviewing hundreds of logs, Phoenix provides a programmatic evaluation framework (phoenix.evals):

  1. QA Correctness: Evaluates whether the generated response accurately answers the user’s question relative to ground-truth answers.
  2. Hallucination & Groundedness (Faithfulness): Compares the generated response strictly against retrieved document chunks:
    • Identifies whether every factual statement in the response is directly supported by the retrieved context.
    • Flags unsupported assertions as hallucinations.
  3. Context Relevance: Analyzes whether the vector database retrieved chunks that actually contain information relevant to the user query, exposing chunking noise and poor embedding alignment.

3. UMAP Latent Embedding Maps: “The Silent Failure Map”

One of Phoenix’s most distinctive capabilities is its latent embedding visualization engine:

  • In production RAG systems, vector retrieval failures are often clustered in specific semantic regions (e.g., users asking questions about billing policies where documentation is missing).
  • Phoenix extracts high-dimensional embedding vectors from queries and documents and runs Uniform Manifold Approximation and Projection (UMAP) to project them into interactive 2D or 3D coordinate space.
  • By overlaying evaluation scores (e.g., coloring points red for hallucination or low relevance), Phoenix visually reveals retrieval blind spots—dense clusters of red points indicate missing documentation or mismatched vector embeddings in the knowledge base.

Hands-On Implementation: Instrumenting and Evaluating a RAG Application

Let’s build a complete, production-grade observability and evaluation workflow using Arize Phoenix. We will launch the Phoenix server, auto-instrument an LLM application with OpenTelemetry, capture traces, and run automated evaluations with LLM-as-a-Judge.

1. Installation & Environment Setup

Install Phoenix and the OpenInference instrumentation libraries:

pip install arize-phoenix arize-phoenix-otel openinference-instrumentation-openai openai pandas

2. Launching Phoenix and Auto-Instrumenting with OpenTelemetry

Launch Phoenix locally and initialize the OpenTelemetry tracer:

import phoenix as px from phoenix.otel import register from openinference.instrumentation.openai import OpenAIInstrumentor from openai import OpenAI # 1. Launch Phoenix in-process observability UI session = px.launch(port=6006) print(f"Phoenix UI active at: {session.url}") # 2. Register OpenTelemetry tracer pointing to local Phoenix collector tracer_provider = register( project_name="enterprise-rag-service", endpoint="http://localhost:6006/v1/traces" ) # 3. Auto-instrument the OpenAI client with OTel GenAI semantic conventions OpenAIInstrumentor().instrument(tracer_provider=tracer_provider) client = OpenAI()

3. Executing Instrumented RAG Operations

Execute a simulated RAG retrieval and response generation. Phoenix automatically captures the entire trace tree in the background:

# Simulated vector retrieval user_query = "What are the in-flight batching guarantees in TensorRT-LLM?" retrieved_chunks = [ "TensorRT-LLM implements in-flight batching (iteration-level scheduling). " "It evicts finished sequences immediately and schedules new prefill requests " "on every token step, eliminating padding token waste." ] # LLM generation with context grounding system_prompt = ( "You are an expert systems engineer. Answer the question using ONLY " f"the provided context:\n\nContext:\n{retrieved_chunks[0]}" ) response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_query} ], temperature=0.2 ) completion_text = response.choices[0].message.content print(f"Generated Output:\n{completion_text}\n")

4. Running Automated LLM-as-a-Judge Evaluations

Extract the captured spans and run an automated evaluation to verify groundedness:

import pandas as pd from phoenix.evals import ( HallucinationEvaluator, QAEvaluator, run_evals, OpenAIModel ) # 1. Initialize evaluation model judge eval_model = OpenAIModel(model="gpt-4o", temperature=0.0) # 2. Define sample evaluation dataset from captured traces eval_dataframe = pd.DataFrame({ "input": [user_query], "context": [retrieved_chunks[0]], "output": [completion_text] }) # 3. Instantiate Hallucination and QA Correctness Evaluators hallucination_evaluator = HallucinationEvaluator(eval_model) qa_evaluator = QAEvaluator(eval_model) # 4. Execute batch evaluation eval_results = run_evals( dataframe=eval_dataframe, evaluators=[hallucination_evaluator, qa_evaluator], provide_explanation=True ) print("\n--- Phoenix Evaluation Results ---") for result in eval_results: print(f"Evaluator: {result.name}") print(f"Score: {result.score} | Label: {result.label}") print(f"Explanation: {result.explanation}\n") # 5. Log evaluation results back to Phoenix UI px.Client().log_evaluations(eval_results) print("Evaluations logged to Phoenix dashboard successfully.")

Production Architecture & Integration for aiFetchly

Integrating Arize Phoenix into automated web scraping pipelines, document extraction workflows, and desktop copilots like AiFetchly establishes a resilient, data-private observability tier.

aiFetchly Enterprise Observability Architecture: ┌─────────────────────────────────────────────────────────────────┐ │ aiFetchly Desktop Application & Web Extraction Workers │ └─────────────────────────────────────────────────────────────────┘ │ ▼ (OTel GenAI Standard Spans via localhost gRPC / HTTP) ┌─────────────────────────────────────────────────────────────────┐ │ Local / Edge Phoenix Observability Daemon (`localhost:6006`) │ │ • Traces multi-step DOM parsing, regex cleaning, and RAG calls │ │ • Evaluates token costs and latency across local & cloud models │ │ • 100% Local Storage: Zero customer prompts leave workstation │ └─────────────────────────────────────────────────────────────────┘ │ ├──► Real-Time Latency & Token Telemetry Dashboard │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Automated CI/CD Regression Testing Gates │ │ • Runs Phoenix Evals on Golden Benchmark Datasets on every git PR│ │ • Blocks release if Groundedness or QA Correctness drops < 95% │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ UMAP Latent Space Analysis (Vector Knowledge Base Optimization) │ │ • Projects embedded document chunks & failed user queries │ │ • Identifies gaps in web scraper coverage for continuous updates│ └─────────────────────────────────────────────────────────────────┘

1. End-to-End Tracing for Autonomous Web Scraping

When AiFetchly executes complex web extraction tasks:

  • Phoenix traces the full lifecycle: from initial HTTP/Playwright web page retrieval, HTML text cleaning, and chunking, to vector database insertion and schema-guided extraction.
  • If a web scraper encounters anti-bot challenges or returns empty DOM elements, Phoenix immediately records the failed span, providing instant root-cause analysis without digging through massive unstructured terminal logs.

2. CI/CD Automated Evaluation Gates

Before releasing new agent prompts or fine-tuned model checkpoints:

  • Run automated test suites against a curated “Golden Dataset” of difficult extraction queries.
  • Phoenix evaluates model outputs for Groundedness, QA Precision, and Schema Validity.
  • If an updated prompt introduces hallucinations or drops below a 95% faithfulness threshold, the CI/CD pipeline fails the build automatically, preventing broken agents from reaching end users.

3. Co-Locating Observability with Local Inference & Storage

For strict enterprise privacy:

  • Deploy Phoenix alongside local LLM runners like Ollama or TensorRT-LLM and embedded vector stores like ChromaDB.
  • Telemetry, embedding coordinates, evaluation scores, and raw prompt completions are stored locally in DuckDB or SQLite, ensuring zero data leakage while delivering enterprise-grade visibility.

Summary: Bringing Scientific Rigor and Observability to Generative AI

Deploying mission-critical artificial intelligence applications requires moving beyond anecdotal testing and black-box guesswork.

By unifying OpenTelemetry distributed tracing, automated LLM-as-a-Judge evaluations, and UMAP latent embedding failure maps into an open-source, self-hosted platform, Arize Phoenix provides the transparency and scientific rigor needed to operationalize generative AI safely. For automated web harvesting, intelligent document extraction, and desktop copilots engineered with AiFetchly, Phoenix delivers the deep observability necessary to eliminate hallucinations, optimize retrieval accuracy, and maintain continuous production reliability.