bge-m3embeddingshybrid-searchragvector-searchtutorial

BGE-M3 Guide: Dense, Sparse & Multi-Vector Hybrid Search

BGE-M3 Architecture: Dense Semantic Vectors, Learned Sparse Weights & ColBERT Multi-Vectors

The Fragmented Retrieval Dilemma in Production RAG

In enterprise search systems, semantic data extraction platforms, and autonomous AI agents powered by AI Fetchly, information retrieval is the bedrock of foundation model reliability. Whether querying unstructured corporate data lakes, financial regulatory filings, or live technical manuals harvested via real-time data fetching, the quality of downstream LLM generation is strictly bounded by the precision and recall of the initial retrieval pass.

For years, information retrieval (IR) practitioners have been forced to choose between three fundamentally competing retrieval paradigms:

The Trilemma of Information Retrieval: ┌────────────────────────────────────────────────────────────────────────┐ │ 1. Dense Semantic Retrieval (Bi-Encoders / Cosine Similarity): │ │ • Strengths: Captures broad semantic intent, synonyms, abstractions │ │ • Weaknesses: Information bottleneck; collapses on rare entities, │ │ exact serial numbers, product SKUs, and specialized jargon │ ├────────────────────────────────────────────────────────────────────────┤ │ 2. Sparse Lexical Retrieval (BM25 / SPLADE / Learned Lexical): │ │ • Strengths: Exact keyword matching; zero semantic drift on numbers │ │ • Weaknesses: Vocabulary mismatch; fails on paraphrasing or concept │ │ queries with zero direct keyword overlap │ ├────────────────────────────────────────────────────────────────────────┤ │ 3. Multi-Vector Late Interaction (ColBERT / MaxSim): │ │ • Strengths: Fine-grained token-to-token soft alignment │ │ • Weaknesses: Massive index memory footprint; slow computation │ │ without complex quantized search engines (PLAID) │ └────────────────────────────────────────────────────────────────────────┘

The Engineering Complexity of Multi-Model Ensembles

To overcome the blind spots of individual paradigms, high-performance enterprise systems historically deployed multi-model hybrid ensembles:

  1. Run BM25 via Elasticsearch or OpenSearch.
  2. Run a dense bi-encoder (such as OpenAI text-embedding-3 or Contriever) via a vector database (like Milvus or Qdrant).
  3. Run a ColBERT model for multi-vector candidate reranking.
  4. Merge candidate lists using Reciprocal Rank Fusion (RRF) or a cross-encoder reranker.

While effective, this ensemble approach introduces a crushing operational tax:

  • Three Separate Indexing Pipelines: Maintaining three distinct database engines, synchronization queues, and embedding models.
  • Excessive Latency and Compute: Generating embeddings three separate times per query and document, tripling memory bus and GPU load.
  • Short Context Bottleneck: Most existing dense and multi-vector models truncate sequences at 512 tokens, forcing aggressive passage chunking that cleaves semantic paragraphs and loses long-range context.

To unify these three disparate paradigms into a single, cohesive, long-context embedding backbone, the Beijing Academy of Artificial Intelligence (BAAI) introduced BGE-M3 (Xiao et al., 2024). BGE-M3 stands for Multi-Lingual (100+ languages), Multi-Functionality (Dense + Sparse + Multi-Vector), and Multi-Granularity (up to 8,192 tokens).

BGE-M3 Architecture: Unifying the Three Retrieval Paradigms

BGE-M3 is built upon a 1024-dimensional transformer encoder (extended from XLM-RoBERTa) that outputs all three retrieval representations in a single forward pass.

BGE-M3 Single-Pass Multi-Functionality Architecture: Input Text (Up to 8,192 Tokens): "What are the Q3 enterprise pricing tiers for AI Fetchly?" │ ▼ ┌────────────────────────────────────────────────────────────────────────┐ │ BGE-M3 Shared Transformer Backbone (XLM-RoBERTa, 560M Parameters): │ │ • Full bidirectional self-attention across 8,192 tokens │ │ • Emits hidden states H = [h_[CLS], h₁, h₂, ..., h_L] ∈ ℝ^(L × 1024) │ └─────────────────────────────────────┬──────────────────────────────────┘ │ ┌────────────────────────────┼────────────────────────────┐ ▼ ▼ ▼ ┌───────────────────┐ ┌───────────────────┐ ┌───────────────────┐ │ 1. Dense Vector │ │ 2. Sparse Lexical │ │ 3. Multi-Vector │ │ • Take h_[CLS] │ │ • Linear Head: │ │ • Token Vectors: │ │ • L2-Normalize │ │ w_t = ReLU(W h) │ │ E_t = L2(W_m h) │ │ • e_dense ∈ ℝ¹⁰²⁴ │ │ • Sparse weights │ │ • Matrix: │ │ │ │ over vocab V │ │ E ∈ ℝ^(L × 1024)│ └───────────────────┘ └───────────────────┘ └───────────────────┘

1. Dense Semantic Embedding ($S_{\text{dense}}$)

The dense embedding represents the global semantic essence of the entire sequence:

  • Extracted directly from the normalized hidden state of the leading [CLS] token:

    $$\vec{e}{\text{dense}} = \frac{h{[\text{CLS}]}}{|h_{[\text{CLS}]}|_2} \in \mathbb{R}^{1024}$$

  • The relevance score between query $q$ and document $d$ is the standard cosine similarity (inner product):

    $$S_{\text{dense}}(q, d) = \vec{e}{q, \text{dense}} \cdot \vec{e}{d, \text{dense}}^\top$$

  • Fast approximate nearest neighbor (ANN) search is executed in sub-5ms using standard vector indexes (HNSW, IVF-Flat, or ScaNN).

2. Learned Sparse Lexical Representation ($S_{\text{sparse}}$)

Unlike traditional BM25 which computes static frequency statistics (TF-IDF), BGE-M3 learns context-dependent lexical importance weights for every token present in the text:

  • Each token hidden state $h_t$ is projected through a lightweight linear layer with a ReLU activation:

    $$w_t = \text{ReLU}\left( W_{\text{lex}} h_t + b_{\text{lex}} \right) \in \mathbb{R}^1$$

  • The sparse vector $\vec{v}_{\text{sparse}} \in \mathbb{R}^{|\mathcal{V}|}$ maps token IDs to their learned positive weights. If a token appears multiple times, its maximum weight is retained:

    $$\vec{v}{\text{sparse}}[t] = \max{i \in \text{Positions}(t)} w_i$$

  • The sparse relevance score is the inner product of the sparse weight vectors (equivalent to matching shared keywords weighted by learned importance):

    $$S_{\text{sparse}}(q, d) = \sum_{t \in q \cap d} \vec{v}{q, \text{sparse}}[t] \cdot \vec{v}{d, \text{sparse}}[t]$$

  • This sparse representation can be exported directly into Lucene, Elasticsearch, or Qdrant sparse vectors, completely eliminating vocabulary mismatch on specialized jargon, serial numbers, and exact product names.

3. Multi-Vector Late Interaction ($S_{\text{multi}}$)

To capture fine-grained token-level alignments without the quadratic latency of cross-encoders, BGE-M3 incorporates the ColBERT-style MaxSim late-interaction operator:

  • Each token hidden state $h_t$ is projected into a normalized multi-vector representation:

    $$E_t = \frac{W_{\text{multi}} h_t}{|W_{\text{multi}} h_t|_2} \in \mathbb{R}^{1024}$$

  • Given query token matrix $E_q \in \mathbb{R}^{|q| \times 1024}$ and document token matrix $E_d \in \mathbb{R}^{|d| \times 1024}$, the multi-vector score evaluates the sum of maximum token similarities:

    $$S_{\text{multi}}(q, d) = \frac{1}{|q|} \sum_{i = 1}^{|q|} \max_{j = 1}^{|d|} \left( E_{q, i} \cdot E_{d, j}^\top \right)$$

  • MaxSim soft-aligns every query token to its most relevant document counterpart, resolving complex relational queries and negation patterns.

Hybrid Retrieval Fusion: Linear Combination & Reciprocal Rank Fusion

Because all three scores ($S_{\text{dense}}$, $S_{\text{sparse}}$, and $S_{\text{multi}}$) are produced from a single model, BGE-M3 allows developers to combine them into an optimized Hybrid Score:

$$S_{\text{hybrid}}(q, d) = \alpha \cdot S_{\text{dense}}(q, d) + \beta \cdot S_{\text{sparse}}(q, d) + \gamma \cdot S_{\text{multi}}(q, d)$$

where $\alpha, \beta, \gamma \ge 0$ are weighting hyperparameters calibrated to the search domain:

  • General Semantic QA: $\alpha = 0.5, \beta = 0.2, \gamma = 0.3$
  • Technical Documentation & Code: $\alpha = 0.3, \beta = 0.4, \gamma = 0.3$
  • Short Entity Search: $\alpha = 0.2, \beta = 0.6, \gamma = 0.2$
The BGE-M3 Two-Stage Production Cascade: Stage 1: Fast Candidate Generation (Top-1000) ┌────────────────────────────────────────────────────────┐ │ Dense Vector ANN Search (HNSW) ──► Top-500 Candidates │ │ ├──► Candidate Pool (Union) │ Sparse Lexical Search (Lucene) ──► Top-500 Candidates │ └────────────────────────────────────────────────────────┘ │ ~1,000 Candidates ▼ Stage 2: Fine-Grained Multi-Vector Reranking (Top-K) ┌────────────────────────────────────────────────────────┐ │ Multi-Vector Late Interaction (MaxSim) │ │ • Dequantize & evaluate MaxSim across 1,000 candidates │ │ • Compute final weighted hybrid score S_hybrid │ │ • Emit Final Top-10 Results to LLM Context │ └────────────────────────────────────────────────────────┘

This two-stage cascade achieves the ultimate search architecture: fast candidate gathering via parallel dense and sparse inverted indexes (<10ms) followed by token-level MaxSim reranking (<15ms), delivering cross-encoder quality at bi-encoder throughput.

The Multi-Task Training Pipeline & Self-Knowledge Distillation

Training a single transformer to excel simultaneously across dense, sparse, and multi-vector objectives is challenging. Standard multi-task learning often suffers from negative gradient interference, where optimizing one objective degrades another.

BAAI solved this with a sophisticated three-stage training curriculum featuring Self-Knowledge Distillation.

BGE-M3 Three-Stage Training Curriculum: ┌────────────────────────────────────────────────────────┐ │ Stage 1: RetroMAE Pre-Training │ │ • Masked Auto-Encoding across 100+ languages │ │ • Establishes high-capacity multilingual representation│ └───────────────────────────┬────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────┐ │ Stage 2: Large-Scale Contrastive Learning (1.2B Pairs) │ │ • Web-scale weakly supervised query-passage pairs │ │ • InfoNCE contrastive loss across Dense, Sparse, Multi │ └───────────────────────────┬────────────────────────────┘ │ ▼ ┌────────────────────────────────────────────────────────┐ │ Stage 3: Fine-Tuning with Self-Knowledge Distillation │ │ • Hard negative mining via cross-encoders │ │ • Ensemble Teacher: S_teacher = (S_d + S_s + S_m) / 3 │ │ • Each individual head learns from the ensemble teacher│ └────────────────────────────────────────────────────────┘

Self-Knowledge Distillation Formulation

During the final fine-tuning phase, BGE-M3 treats the ensemble of its own three retrieval heads as a teacher model.

For a batch of queries and candidate passages, the three individual similarity scores $S_{\text{dense}}$, $S_{\text{sparse}}$, and $S_{\text{multi}}$ are averaged to form an ensemble teacher distribution:

$$S_{\text{teacher}}(q, d) = \frac{1}{3} \left( S_{\text{dense}}(q, d) + S_{\text{sparse}}(q, d) + S_{\text{multi}}(q, d) \right)$$

The target probability distribution over candidates is:

$$P_{\text{teacher}}(d_i | q) = \frac{\exp(S_{\text{teacher}}(q, d_i) / \tau)}{\sum_j \exp(S_{\text{teacher}}(q, d_j) / \tau)}$$

Each individual head ($m \in {\text{dense}, \text{sparse}, \text{multi}}$) is optimized to minimize the Kullback-Leibler (KL) divergence against the teacher distribution alongside the ground-truth contrastive loss:

$$\mathcal{L}{\text{total}} = \sum{m} \left( \mathcal{L}{\text{InfoNCE}}^m + \lambda{\text{KD}} \cdot D_{\text{KL}}\left( P_{\text{teacher}} ;\middle|; P_m \right) \right)$$

Through Self-Knowledge Distillation:

  • The dense head learns exact keyword discrimination from the sparse head.
  • The sparse head learns semantic generalization from the dense head.
  • The multi-vector head refines token alignment from the ensemble consensus.

Negative gradient interference is eliminated, and all three retrieval heads achieve higher individual performance than if trained in isolation!

End-to-End Implementation: Using BGE-M3 in Python

In this section, we implement a complete BGE-M3 hybrid retrieval workflow using the official FlagEmbedding library and native PyTorch.

1. Installation

Install FlagEmbedding and PyTorch:

pip install FlagEmbedding torch transformers

2. Hybrid Embedding Extraction & Scoring

from FlagEmbedding import BGEM3FlagModel import numpy as np # 1. Initialize BGE-M3 Model (FP16 on GPU) model = BGEM3FlagModel('BAAI/bge-m3', use_fp16=True) # 2. Define Query and Passages query = "What is the memory footprint of ColBERTv2 compared to standard dense vectors?" documents = [ "ColBERTv2 compresses multi-vector indexes by 8x using residual vector quantization, requiring 48 GB for 10M documents.", "Dense bi-encoders encode each passage into a single vector, requiring 41 GB in FP16 precision.", "AI Fetchly provides high-throughput automated web scraping pipelines for enterprise RAG applications.", "The weather in Tokyo is mild and temperate during the autumn months." ] # 3. Extract All Three Embeddings Simultaneously query_output = model.encode( query, return_dense=True, return_sparse=True, return_colbert_vecs=True ) doc_output = model.encode( documents, return_dense=True, return_sparse=True, return_colbert_vecs=True ) # 4. Compute Individual and Hybrid Relevance Scores print("--- Evaluating BGE-M3 Hybrid Retrieval Scores ---") for idx, doc_text in enumerate(documents): # Compute Dense Score (Cosine Similarity) dense_score = np.dot(query_output['dense_vecs'], doc_output['dense_vecs'][idx]) # Compute Sparse Lexical Score (Shared Term Weights) sparse_score = model.compute_lexical_matching_score( query_output['lexical_weights'], doc_output['lexical_weights'][idx] ) # Compute Multi-Vector Late Interaction (MaxSim) colbert_score = model.colbert_score( query_output['colbert_vecs'], doc_output['colbert_vecs'][idx] ) # Weighted Hybrid Fusion hybrid_score = 0.4 * dense_score + 0.2 * sparse_score + 0.4 * colbert_score print(f"\nDocument {idx+1}: {doc_text[:80]}...") print(f" Dense Score: {dense_score:.4f}") print(f" Sparse Score: {sparse_score:.4f}") print(f" ColBERT Score: {colbert_score:.4f}") print(f" Hybrid Score: {hybrid_score:.4f}")

3. PyTorch Reference Implementation of BGE-M3 Output Heads

The following PyTorch module demonstrates how the three output heads are constructed on top of a shared transformer hidden representation:

import torch import torch.nn as nn import torch.nn.functional as F from typing import Dict, Tuple class BGEM3OutputHeads(nn.Module): """ Production-grade reference implementation of BGE-M3 multi-functionality output heads. Projects shared transformer hidden states into Dense, Sparse Lexical, and Multi-Vector representations. """ def __init__(self, hidden_dim: int = 1024): super().__init__() self.hidden_dim = hidden_dim # Dense projection (operates on [CLS]) self.dense_proj = nn.Linear(hidden_dim, hidden_dim) # Sparse lexical linear head: maps hidden state to a positive term weight scalar self.sparse_linear = nn.Linear(hidden_dim, 1) # Multi-vector linear head: maps each token to a normalized ColBERT vector self.multi_proj = nn.Linear(hidden_dim, hidden_dim) def forward( self, hidden_states: torch.Tensor, attention_mask: torch.Tensor ) -> Dict[str, torch.Tensor]: """ Args: hidden_states: Tensor of shape (batch, seq_len, hidden_dim) attention_mask: Binary mask of shape (batch, seq_len) Returns: Dictionary containing 'dense', 'sparse_weights', and 'multi_vectors' """ # 1. Dense Representation: Extract [CLS] token and normalize cls_token = hidden_states[:, 0, :] dense_vecs = F.normalize(self.dense_proj(cls_token), p=2, dim=-1) # 2. Sparse Lexical Weights: w_t = ReLU(W · h_t) # Shape: (batch, seq_len) sparse_weights = F.relu(self.sparse_linear(hidden_states)).squeeze(-1) # Zero out padding tokens sparse_weights = sparse_weights * attention_mask.float() # 3. Multi-Vector Representations: ColBERT-style normalized vectors # Shape: (batch, seq_len, hidden_dim) multi_vecs = F.normalize(self.multi_proj(hidden_states), p=2, dim=-1) # Mask out padding tokens multi_vecs = multi_vecs * attention_mask.unsqueeze(-1).float() return { "dense": dense_vecs, "sparse_weights": sparse_weights, "multi_vectors": multi_vecs } # Verification Unit Test if __name__ == "__main__": batch_size = 2 seq_len = 32 hidden_dim = 1024 heads = BGEM3OutputHeads(hidden_dim=hidden_dim) h_mock = torch.randn(batch_size, seq_len, hidden_dim) mask_mock = torch.ones(batch_size, seq_len, dtype=torch.long) mask_mock[:, 24:] = 0 # Simulate padding outputs = heads(h_mock, mask_mock) print("Dense Vectors Shape: ", outputs["dense"].shape) print("Sparse Token Weights Shape: ", outputs["sparse_weights"].shape) print("Multi-Vectors Shape: ", outputs["multi_vectors"].shape) assert outputs["dense"].shape == (batch_size, hidden_dim) assert outputs["sparse_weights"].shape == (batch_size, seq_len) assert outputs["multi_vectors"].shape == (batch_size, seq_len, hidden_dim) print("BGE-M3 Multi-Head Forward Pass Validated Successfully!")

When managing high-volume data harvesting with our automated web scraping tools, passing extracted web documents directly into BGE-M3 via our custom API integration allows indexing structured sparse keywords and dense semantic vectors in a single pipeline pass.

Empirical Benchmarks & Performance Analysis

In official benchmark evaluations published by BAAI and verified on the MTEB (Massive Text Embedding Benchmark) and MIRACL (Multilingual Information Retrieval Across a Continuum of Languages) benchmarks, BGE-M3 demonstrated state-of-the-art retrieval accuracy across multilingual and multi-task datasets.

Multi-Domain Retrieval Evaluation

The following table summarizes retrieval accuracy (nDCG@10) across diverse domains comparing Dense BM25, OpenAI text-embedding-3, Cohere, and BGE-M3:

Model ArchitectureRepresentationsMTEB Retrieval Avg (nDCG@10)MIRACL Multilingual (nDCG@10)Long-Context BioASQ (8k Tokens)Keyword Heavy Legal Bench
BM25 / AnseriniSparse Lexical0.4280.4520.3840.495
OpenAI text-embedding-3-largeDense Only (3072-d)0.5540.5820.4860.512
Cohere Embed-v3Dense Only (1024-d)0.5580.6100.4920.528
ColBERTv2Multi-Vector (128-d)0.5280.5400.441 (Truncated)0.554
BGE-M3 (Dense Only)Dense (1024-d)0.5620.6240.5120.535
BGE-M3 (Sparse Only)Learned Sparse0.4850.5210.4620.578
BGE-M3 (Multi-Vector)ColBERT (1024-d)0.5690.6480.5340.589
BGE-M3 (Hybrid Fused)Dense + Sparse + Multi0.598 (SOTA)0.672 (SOTA)0.568 (SOTA)0.624 (SOTA)

Critical Empirical Insights

  1. The Synergy of Hybrid Fusion: While individual heads perform competitively with commercial models, fusing Dense + Sparse + Multi-Vector yields a massive 3.6 percentage point gain on MTEB (0.598 vs 0.562). The sparse head fixes lexical misses on legal and technical queries, while the dense head anchors high-level semantic abstractions.
  2. True 8,192 Token Context Mastery: On long-document retrieval benchmarks (such as BioASQ and MultiFieldQA), standard 512-token models suffer severe truncation degradation. BGE-M3 maintains high accuracy because its full 8,192-token attention window encodes entire technical manuscripts without chunk fragmentation.
  3. Multilingual Superiority on 100+ Languages: On MIRACL (spanning 18 high- and low-resource languages), BGE-M3 Hybrid achieves 0.672, outperforming Cohere Embed-v3 (0.610) and OpenAI text-embedding-3 (0.582) by substantial margins.

Production Engineering: Database Topologies & Serving Patterns

Deploying BGE-M3 in enterprise production environments requires selecting the appropriate database storage topology to balance retrieval latency, RAM consumption, and search precision.

Enterprise Hybrid Storage Architecture with BGE-M3: ┌─────────────────────────┐ ┌───────────────────────────────┐ │ Extracted Web / Doc Data│ ──► │ BGE-M3 High-Throughput Worker │ └─────────────────────────┘ └──────────────┬────────────────┘ │ Single Forward Pass ┌─────────────────────────────┼─────────────────────────────┐ ▼ ▼ ▼ ┌─────────────────────────────────┐ ┌─────────────────────────┐ ┌─────────────────────────┐ │ Qdrant / Milvus (Dense HNSW) │ │ Elasticsearch / OpenSearch│ │ Object Storage / NVMe │ │ • Stores e_dense (1024-d) │ │ • Stores sparse weights │ │ • Stores multi-vectors │ │ • Sub-5ms ANN Candidate Search │ │ • Exact keyword inverted │ │ • Used only for final │ │ (Retrieves Top-500) │ │ index search │ │ Top-100 ColBERT rerank│ └─────────────────────────────────┘ └─────────────────────────┘ └─────────────────────────┘

1. Unified Hybrid Storage in Modern Vector Databases

Modern vector databases (such as Qdrant, Milvus 2.4+, and Pinecone) natively support both dense and sparse vectors in the same record:

  • Store BGE-M3’s dense vector in the dense index (HNSW).

  • Store BGE-M3’s learned lexical weights in the sparse index (Inverted Index).

  • At query time, issue a single hybrid query using Reciprocal Rank Fusion (RRF):

    $$\text{Score}{\text{RRF}}(d) = \frac{1}{60 + \text{Rank}{\text{dense}}(d)} + \frac{1}{60 + \text{Rank}_{\text{sparse}}(d)}$$

This yields 95% of full hybrid accuracy in under 12ms without requiring multi-vector storage!

2. Selective Multi-Vector Reranking

Multi-vectors consume significant storage ($8,192 \text{ tokens} \times 1024 \text{ dims} \times 2 \text{ bytes} \approx 16 \text{ MB per document}$).

  • Do not store uncompressed multi-vectors for millions of documents in RAM.
  • Store multi-vectors compressed on disk or compute them dynamically on the top-50 candidates returned by the fast dense-sparse stage.
  • Alternatively, apply residual vector quantization (similar to ColBERTv2) to compress multi-vectors by 8x.

3. Batched Embedding Inference with ONNX and TensorRT

Because BGE-M3 uses a 560M parameter encoder:

  • Convert the model to TensorRT or ONNX Runtime with FP16 precision.
  • On an NVIDIA L4 or A10G GPU, TensorRT achieves over 1,200 sentences per second throughput.
  • Enable dynamic padding so short queries are not padded to 8,192 tokens during embedding generation.

Review our flexible pricing tiers for compute options designed for high-scale enterprise embedding and retrieval workloads.

For comprehensive architectural specifications, integration blueprints, and deployment checklists, review our comprehensive developer documentation.

Frequently Asked Questions (FAQ)

What does “M3” stand for in BGE-M3?

“M3” stands for the three core capabilities of the model: Multi-Linguality (supporting over 100 spoken and programming languages), Multi-Functionality (supporting Dense, Learned Sparse, and Multi-Vector representations simultaneously), and Multi-Granularity (handling inputs from short sentence queries up to 8,192-token full documents).

How does BGE-M3 sparse retrieval differ from BM25?

BM25 relies on static, surface-level string matching based on term frequency (TF) and inverse document frequency (IDF). It treats all words with the same frequency equally and cannot adjust for context. BGE-M3’s sparse representation uses a neural head to output learned importance weights: it understands context, penalizes common filler words, and boosts critical domain-specific entities even if they appear only once in the text.

Yes. BGE-M3’s dense head can be used as a standalone 1024-dimensional dense embedding model. Even without sparse or multi-vector fusion, BGE-M3’s dense head outperforms many dedicated dense embedding models due to the regularizing benefits of multi-task pre-training and self-knowledge distillation.

What is the maximum sequence length supported by BGE-M3?

BGE-M3 natively supports context windows up to 8,192 tokens. This allows encoding entire academic papers, legal agreements, and technical RFC specifications without arbitrary chunking or semantic fragmentation.