milvusmilvus-litevector-databasefull-text-searchhybrid-searchragtutorial

Milvus 2.5 Guide: BM25 Full-Text Search & Milvus Lite

Milvus 2.5 Architecture: Native Sparse-BM25 Full-Text Search, Embedded Milvus Lite, Multi-Vector Hybrid RRF Reranking, and Disaggregated Cloud Scale

The Problem: The Dual-Database Tax and Prototyping Gap in Enterprise Vector Retrieval

In modern artificial intelligence engineering, automated web data extraction systems, and desktop knowledge copilots powered by AiFetchly, foundation models rely on Retrieval-Augmented Generation (RAG) to access live, private enterprise data. Real-world enterprise datasets are complex: they contain dense narrative paragraphs, tabular financial data, domain-specific acronyms, exact serial numbers, and programming code snippets.

However, deploying enterprise-grade retrieval has historically forced engineering teams to confront two punishing architectural dilemmas: the Dual-Database Tax and the Prototyping Gap.

The Enterprise Retrieval Architecture Bottleneck: ┌─────────────────────────────────────────────────────────────────┐ │ User Query: "Find compliance breach report for SKU XG-9021" │ └─────────────────────────────────────────────────────────────────┘ │ ▼ [The Traditional Dual-Database Architecture] ├── Database 1: Elasticsearch / OpenSearch Cluster (BM25 Keyword)│ │ ├── Separate JVM cluster, expensive cloud RAM & disk │ │ └── Matches exact serial "XG-9021", but misses semantics │ │ │ └── Database 2: Standalone Vector Database (Dense Embeddings) │ ├── High-maintenance Docker/K8s cluster, separate API keys │ └── Understands "compliance breach", but ignores SKU number! │ │ ▼ [Application-Level Glue Code & Reranking] ├── ❌ Synchronization Lag: Dual-write race conditions & stale indexes! ├── ❌ Operational Overhead: Managing two distributed databases! ├── ❌ Prototyping Barrier: Developers cannot run heavy JVM/K8s locally! └── ❌ Network Roundtrip Penalty: Multiple hops inflate retrieval latency!

1. The Dual-Database Tax: Maintaining Elasticsearch Alongside Vector Engines

To achieve high-precision retrieval across both semantic concepts and exact keywords, traditional RAG architectures forced developers to run two completely separate databases in parallel:

  • Elasticsearch or OpenSearch for sparse lexical keyword matching (BM25).
  • A dedicated vector database (such as early Milvus, Pinecone, or Qdrant) for dense semantic vector search.
  • The Synchronization Nightmare: Keeping two independent databases in sync requires complex distributed transactions, dual-write queues, or Change Data Capture (CDC) pipelines. If a document is updated in one store but delayed in the other, retrieval queries return contradictory results.
  • Cost Multiplier: Enterprises pay double for infrastructure: hosting separate compute nodes, memory buffers, and backup storage for two distinct distributed systems.

2. The Prototyping Gap: Heavy Infrastructure vs. Embedded Simplicity

Distributed vector databases were traditionally architected exclusively as complex distributed clusters:

  • Deploying a full cluster required orchestrating Kubernetes pods, message brokers (Kafka or Pulsar), object storage (MinIO or S3), and etcd coordinators.
  • Developer Friction: A software engineer prototyping a desktop application, writing unit tests in CI/CD, or experimenting in a local Jupyter notebook could not easily launch a multi-node cluster.
  • When teams used lightweight in-memory vector stores (like basic FAISS or NumPy) for local development, they encountered severe API divergence when migrating to production, requiring complete rewrites of data-access layers.

3. The Multi-Vector Complexity Barrier

Modern multimodal and hybrid retrieval requires indexing entities across multiple vector spaces simultaneously:

  • A single product listing might possess a dense text embedding, a sparse BM25 keyword vector, and a dense visual image embedding.
  • In legacy systems, searching across multiple vector representations required executing separate queries against different endpoints and implementing custom rank-fusion algorithms in Python application code, introducing network latency and maintenance overhead.

The Solution: Milvus 2.5 and Milvus Lite Unified Architecture

To eliminate the Dual-Database Tax and bridge the gap between local prototyping and planetary-scale deployment, Zilliz and the open-source community created Milvus 2.5.

Milvus 2.5 is an open-source, cloud-native vector database written in Go and C++. Built around the high-performance Knowhere vector execution core, Milvus 2.5 natively unifies Sparse-BM25 full-text search, multi-vector hybrid search with built-in Reciprocal Rank Fusion (RRF), and Milvus Lite—a zero-dependency, in-process Python engine that runs directly inside client applications without external daemons.

Milvus 2.5 Unified Architecture: ┌─────────────────────────────────────────────────────────────────┐ │ Client Applications (aiFetchly Desktop, Python Scripts, APIs) │ └─────────────────────────────────────────────────────────────────┘ │ ▼ (Unified pymilvus MilvusClient API) ┌─────────────────────────────────────────────────────────────────┐ │ Deployment Tier: Embedded or Distributed Cluster │ │ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Milvus Lite (Embedded In-Process Engine) │ │ │ │ • Zero Docker / daemons: `pip install pymilvus` │ │ │ │ • Local `.db` file storage backed by embedded Knowhere C++ │ │ │ └─────────────────────────────────────────────────────────────┘ │ │ ▲ │ │ 100% Identical Client API & Schema Parity! │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────┐ │ │ │ Milvus 2.5 Distributed (Storage-Compute Disaggregation) │ │ │ │ • Stateless Worker Nodes: Query Nodes, Data Nodes, Index │ │ │ │ • Streaming Broker: Apache Kafka / Pulsar event stream │ │ │ │ • Persistent Storage: MinIO / Amazon S3 Object Storage │ │ │ └─────────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Knowhere C++ Unified Search Engine Core │ │ • Native Sparse-BM25 Inverted Index (Zero Elasticsearch!) │ │ • Dense Vector Indexing: HNSW, IVF-Flat, and DiskANN │ │ • Multi-Vector Hybrid Search with Reciprocal Rank Fusion (RRF) │ └─────────────────────────────────────────────────────────────────┘ │ ▼ (Sub-10ms Ranked Retrieval) ┌─────────────────────────────────────────────────────────────────┐ │ Ranked Top-K Chunks Grounded with Dense & Lexical Precision │ └─────────────────────────────────────────────────────────────────┘

The Architectural Pillars of Milvus 2.5

  1. Native Sparse-BM25 Full-Text Search: Milvus 2.5 embeds text tokenization, vocabulary dictionary construction, and inverted BM25 term frequency indexing directly into its C++ Knowhere core. Developers ingest raw text; Milvus computes sparse term frequencies internally, completely eliminating the need for an external Elasticsearch cluster.
  2. Milvus Lite (Zero-Dependency In-Process Mode): Included directly inside pymilvus. Running MilvusClient("./milvus.db") spins up an embedded vector engine inside the host Python process, writing to a single local database file with zero Docker, network configuration, or background daemon requirements.
  3. Seamless Scale Transition (Zero Code Changes): The exact same MilvusClient code that runs locally on a developer workstation via Milvus Lite can point to a massive multi-node distributed cluster (MilvusClient("http://milvus-cluster:19530")) simply by updating the connection URI string.
  4. Multi-Vector Hybrid Search & Native Reranking: Enables collections with multiple vector fields per entity (e.g., dense text vector + sparse BM25 vector + image embedding). The query engine searches all vector spaces in parallel and executes Reciprocal Rank Fusion (RRF) or weighted score fusion natively in C++ before returning results.
  5. Disaggregated Storage-Compute Architecture: Stateless worker nodes (Query Nodes for read, Data Nodes for write, Index Nodes for graph construction) scale independently, backed by Kafka/Pulsar streaming write-ahead logs and S3-compatible cloud object storage.

Comparing Modern Enterprise Vector Search Engines

The table below contrasts Milvus 2.5 against other leading vector and hybrid retrieval systems:

Vector Search & Hybrid Retrieval Architectural Comparison: ┌───────────────────────────┬───────────────────────────────┬───────────────────────────────┬───────────────────────────────┐ │ Dimension │ Elasticsearch + Vector Plugin │ Pinecone │ Milvus 2.5 & Milvus Lite │ ├───────────────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┤ │ Core Implementation │ Java (Lucene inverted index) │ Closed-source proprietary │ Go (Coordinators) + C++ (Core)│ │ Embedded In-Process Mode │ No (Requires JVM server) │ No (Cloud SaaS only) │ Yes (Milvus Lite via Python) │ │ Full-Text Search (BM25) │ Native (Original core) │ Sparse vector embedding APIs │ Native Sparse-BM25 in C++ core│ │ Multi-Vector Support │ Complex manual script queries │ Namespaces & single vector │ Native Multi-Vector + RRF │ │ Disk-Backed Indexing │ Lucene memory-mapped files │ Proprietary serverless tier │ DiskANN & In-Memory HNSW │ │ Billion-Scale Scaling │ Heavy JVM cluster maintenance │ Automated managed cloud │ Disaggregated Storage-Compute │ │ Self-Hosting Capability │ Yes (Self-managed) │ No (Cloud only) │ Yes (100% Open Source Apache) │ └───────────────────────────┴───────────────────────────────┴───────────────────────────────┴───────────────────────────────┘

Deep Dive into Milvus 2.5 Core Primitives & Engines

Milvus 2.5 achieves enterprise throughput and developer simplicity through four foundational subsystems: Milvus Lite, Native Sparse-BM25, Multi-Vector RRF Reranking, and Disaggregated Cloud Scaling.

Milvus 2.5 Core Subsystems: Milvus 2.5 Engine ├──► 1. Milvus Lite: In-process embedded Python runtime with single-file `.db` storage ├──► 2. Sparse-BM25: Native full-text tokenization and inverted index inside Knowhere C++ ├──► 3. Multi-Vector RRF: Parallel multi-space search fused via Reciprocal Rank Fusion ├──► 4. Knowhere C++: Vector acceleration supporting HNSW, IVF-Flat, and DiskANN └──► 5. Disaggregated Cluster: Stateless Query/Data/Index workers with Kafka & MinIO

1. Milvus Lite: The In-Process Embedded Engine

Milvus Lite packages the core vector search engine directly into the pymilvus Python distribution:

  • No Background Daemons: It runs entirely within the address space of the host Python process.
  • Embedded C++ Knowhere: Vector calculations and graph traversals are executed via native C++ bindings, delivering the exact same sub-10ms retrieval performance as the distributed server.
  • Single-File Persistence: All collections, vector embeddings, inverted indices, and scalar metadata are stored in a single local database file (./milvus_demo.db), making it ideal for desktop applications, CI/CD testing, and offline edge tools.

2. Native Sparse-BM25 Full-Text Search

In Milvus 2.5, full-text search is no longer an afterthought requiring external services:

  • Automatic Tokenization: When a collection schema includes a VARCHAR text field, Milvus 2.5 attaches a built-in text analyzer that performs lowercasing, punctuation stripping, and tokenization.
  • Sparse Vector Generation: The analyzer converts raw document strings into sparse vector dictionaries representing BM25 term frequencies ($TF$) and inverted document frequencies ($IDF$): $$\text{BM25}(q, d) = \sum_{t \in q} \text{IDF}(t) \cdot \frac{\text{TF}(t, d) \cdot (k_1 + 1)}{\text{TF}(t, d) + k_1 \cdot \left(1 - b + b \cdot \frac{|d|}{\text{avgdl}}\right)}$$
  • Unified Query Routing: When a user queries text, Milvus tokenizes the search string and queries the sparse inverted index directly within the same forward pass as dense vector indices.

3. Multi-Vector Hybrid Search & Reciprocal Rank Fusion (RRF)

Collections in Milvus 2.5 can define multiple distinct vector fields within the same schema:

  • Parallel Sub-Searches: A single query can submit a dense text embedding to dense_vector, a sparse BM25 query to sparse_vector, and an image feature vector to visual_vector.
  • C++ Reciprocal Rank Fusion (RRF): Instead of forcing developers to write normalization heuristics for incompatible distance scores, Milvus fuses candidate rankings directly inside its C++ engine: $$\text{RRF_Score}(d) = \sum_{i=1}^{M} \frac{1}{k + \text{rank}_i(d)}$$ Where $k$ is a smoothing constant (typically $60$) and $\text{rank}_i(d)$ is the document’s position in sub-search $i$.

4. Disaggregated Cloud-Native Architecture

For production deployments managing billions of vectors, Milvus 2.5 scales across four independent layers:

  1. Access Layer: Stateless Proxies handle client gRPC requests, validate schemas, and route operations.
  2. Coordinator Layer: Manages cluster topology and metadata assignments (RootCoord, DataCoord, QueryCoord).
  3. Worker Layer: Stateless computing workers execute specific workloads: QueryNode executes vector and BM25 searches; DataNode consumes write-ahead logs; IndexNode constructs HNSW and DiskANN graphs.
  4. Storage Layer: Decouples persistent storage into Apache Kafka / Pulsar for real-time streaming ingestion and MinIO / Amazon S3 for long-term historical vector segments.

Hands-On Implementation: Building a Hybrid Search Engine with Milvus Lite

Let’s build a complete, production-grade hybrid retrieval engine using Milvus Lite and Python. We will configure an embedded client, define a schema with native Sparse-BM25 full-text search and dense vector embeddings, insert documents, and execute multi-vector hybrid queries with RRF reranking.

1. Installation & Environment Prerequisites

Install pymilvus (which includes Milvus Lite out of the box) and model embedding utilities:

pip install -U pymilvus pymilvus[model]

2. Initializing Milvus Lite & Defining the Multi-Vector Schema

Initialize the embedded client and define a collection supporting both dense vectors and native BM25 full-text search:

from pymilvus import ( MilvusClient, DataType, Function, FunctionType, AnnSearchRequest, RRFRanker ) # 1. Initialize embedded Milvus Lite client (single-file local persistence) client = MilvusClient("./enterprise_knowledge.db") print("Initialized Milvus Lite client pointing to ./enterprise_knowledge.db") # 2. Define Collection Schema schema = client.create_schema( auto_id=True, enable_dynamic_field=True ) # Add primary key and raw text field schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True) schema.add_field(field_name="text", datatype=DataType.VARCHAR, max_length=2048, enable_analyzer=True) schema.add_field(field_name="category", datatype=DataType.VARCHAR, max_length=64) # Add Dense Vector Field (e.g., 384-dimensional MiniLM) schema.add_field(field_name="dense_vector", datatype=DataType.FLOAT_VECTOR, dim=384) # Add Sparse Vector Field for Native BM25 Full-Text Search schema.add_field(field_name="sparse_bm25", datatype=DataType.SPARSE_FLOAT_VECTOR) # 3. Add Built-in BM25 Function (Converts raw text to sparse vector automatically!) bm25_func = Function( name="text_bm25_fn", function_type=FunctionType.BM25, input_field_names=["text"], output_field_names=["sparse_bm25"] ) schema.add_function(bm25_func) # 4. Create Index Parameters index_params = client.prepare_index_params() index_params.add_index( field_name="dense_vector", index_type="AUTOINDEX", metric_type="COSINE" ) index_params.add_index( field_name="sparse_bm25", index_type="SPARSE_INVERTED_INDEX", metric_type="BM25" ) # 5. Create Collection collection_name = "enterprise_rag_kb" client.create_collection( collection_name=collection_name, schema=schema, index_params=index_params ) print(f"Collection '{collection_name}' initialized with Native BM25 and Dense Vectors.")

3. Ingesting Documents with Automatic BM25 Tokenization

Insert raw document text and pre-computed dense embeddings. Milvus Lite tokenizes the text and populates the sparse BM25 inverted index automatically:

import numpy as np # Sample technical documents documents = [ {"text": "TensorRT-LLM accelerates GPU inference using in-flight continuous batching.", "category": "inference"}, {"text": "Milvus 2.5 integrates native Sparse-BM25 full-text search directly into Knowhere C++.", "category": "database"}, {"text": "ChromaDB provides an in-process vector database with SQLite metadata storage.", "category": "database"}, {"text": "Security report: Critical patch required for database driver CVE-2025-64513 on port 19530.", "category": "security"}, {"text": "Weaviate implements native multi-tenancy with dynamic S3 cold-storage offloading.", "category": "database"} ] # Generate synthetic 384-dim dense embeddings (or use SentenceTransformers) rng = np.random.default_rng(42) for doc in documents: doc["dense_vector"] = rng.random(384, dtype=np.float32).tolist() # Insert data into collection res = client.insert( collection_name=collection_name, data=documents ) print(f"Successfully inserted {res['insert_count']} documents into Milvus Lite.")

4. Executing Multi-Vector Hybrid Search with Reciprocal Rank Fusion (RRF)

Execute a hybrid search query combining exact serial keyword matching with dense semantic similarity:

# 1. Define query text and query dense vector search_text = "Critical security vulnerability patch CVE-2025-64513" search_dense_vector = rng.random(384, dtype=np.float32).tolist() # 2. Build Dense Sub-Search Request req_dense = AnnSearchRequest( data=[search_dense_vector], anns_field="dense_vector", param={"metric_type": "COSINE"}, limit=5 ) # 3. Build Native Sparse-BM25 Sub-Search Request req_sparse = AnnSearchRequest( data=[search_text], anns_field="sparse_bm25", param={"metric_type": "BM25"}, limit=5 ) # 4. Execute Multi-Vector Hybrid Search with Native Reciprocal Rank Fusion (RRF) hybrid_results = client.hybrid_search( collection_name=collection_name, reqs=[req_dense, req_sparse], ranker=RRFRanker(k=60), limit=2, output_fields=["text", "category"] ) print("\n--- Milvus 2.5 Multi-Vector Hybrid Results ---") for hit in hybrid_results[0]: print(f"ID: {hit['id']} | RRF Score: {hit['distance']:.4f}") print(f"Category: {hit['entity']['category']}") print(f"Text Snippet: {hit['entity']['text']}\n")

5. Transitioning from Milvus Lite to a Distributed Cluster

When your application outgrows a single machine and scales to a multi-node Kubernetes cluster:

# To transition from local prototype to distributed enterprise cluster, # simply change the connection URI. All schema and query logic remain 100% identical! # Embedded Local Prototype: # client = MilvusClient("./enterprise_knowledge.db") # Production Distributed Cluster: client = MilvusClient( uri="http://milvus-proxy.internal.net:19530", token="root:MilvusPassword123" )

Production Architecture & Integration for aiFetchly

Integrating Milvus 2.5 into automated web harvesting pipelines, enterprise SaaS platforms, and desktop copilots like AiFetchly establishes a resilient, dual-mode retrieval tier.

aiFetchly Enterprise Vector Retrieval Architecture: ┌─────────────────────────────────────────────────────────────────┐ │ aiFetchly Web Scrapers & Distributed Document Harvesters │ └─────────────────────────────────────────────────────────────────┘ │ ├── Mode 1: Client Desktop Application (aiFetchly Local Mode) │ └── Embedded Milvus Lite (`./user_vault.db`) inside Electron │ • Zero Docker / daemons, 100% offline privacy │ └── Mode 2: Multi-Tenant Enterprise Cloud Ingestion └── Milvus 2.5 Distributed Cluster (Disaggregated K8s) • Ingests 50,000+ scraped web pages/sec via Kafka │ ▼ (Parallel Dense Semantic Search + Native Sparse-BM25) ┌─────────────────────────────────────────────────────────────────┐ │ Knowhere C++ Engine Core: Multi-Vector Reciprocal Rank Fusion │ │ • Blends exact technical SKU matching with conceptual retrieval │ └─────────────────────────────────────────────────────────────────┘ │ ├──► Sub-10ms Grounded Context Passed to Inference Tier │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Local / Cloud Foundation Models (Ollama, TensorRT-LLM, Claude) │ └─────────────────────────────────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Observability & Telemetry via OpenTelemetry (Arize Phoenix) │ │ • Traces hybrid search latency, RRF weights, and citation precision│ └─────────────────────────────────────────────────────────────────┘

1. Unified Client API Across Desktop and Cloud

When AiFetchly runs as a local Electron desktop application:

  • Milvus Lite runs directly in-process via Python bindings.
  • Users store private document vectors, scraped websites, and notes in a local .db file with zero configuration.
  • When enterprise customers upgrade to team-wide collaborative workspaces, the desktop app switches its connection URI to the organization’s central Milvus 2.5 cluster without modifying a single line of search query code.

2. Native BM25 for Precise Document Scraping Retrieval

In automated web harvesting:

  • Scraped web documents frequently contain technical identifiers, product codes, and legal terms that dense embedding models fail to resolve.
  • Milvus 2.5’s native Sparse-BM25 full-text search indexes keywords automatically upon insertion, eliminating the need to maintain an auxiliary Elasticsearch cluster and cutting cloud infrastructure bills by over 50%.

3. Full-Stack Tracing and Evaluation

Connect Milvus 2.5 directly to modern observability platforms:

  • Instrument hybrid retrieval operations with Arize Phoenix using OpenTelemetry semantic conventions to track sub-search latencies.
  • Continuously evaluate hybrid retrieval precision using DeepEval or RAGAS to optimize RRF $k$-parameters and dense-to-sparse score weighting.

Summary: Unifying Lexical and Semantic Search from Edge to Cloud

Modern enterprise Retrieval-Augmented Generation requires infrastructure that bridges the gap between local developer agility and planetary-scale cloud throughput.

By unifying native Sparse-BM25 full-text search directly within its C++ Knowhere engine, providing Milvus Lite for zero-dependency in-process Python execution, and delivering multi-vector hybrid search with native Reciprocal Rank Fusion, Milvus 2.5 eliminates the Dual-Database Tax and simplifies production AI architectures. For automated web harvesting, intelligent document extraction, and desktop copilots engineered with AiFetchly, Milvus 2.5 provides the versatile, high-precision retrieval foundation necessary to scale from local prototypes to billion-vector enterprise knowledge bases.