infinityembeddingsrerankingragmodel-servingflashattentiontensorrttutorial

Infinity Guide: High-Throughput Embeddings & Reranking

Infinity Architecture: High-Throughput Embeddings and Reranking Engine with Dynamic Batching, FlashAttention-2 Unpadding, and TensorRT Execution Providers

The Problem: The Inefficiency and Latency Tax of Naive Embedding Serving

In modern enterprise automation, automated web scraping platforms, and real-time desktop copilots like AiFetchly, Retrieval-Augmented Generation (RAG) is the foundational architecture used to ground foundation models in verified facts. Whether indexing live corporate documentation, harvested e-commerce catalogs, or financial disclosures, high-dimensional vector embeddings and cross-encoder rerankers determine retrieval accuracy.

While software engineering teams invest heavily in optimizing generative LLM inference engines (deploying vLLM, SGLang, or TensorRT-LLM for generation), the upstream retrieval pipeline—generating vector embeddings and reranking candidate passages—is frequently served using naive Hugging Face pipelines or custom FastAPI wrapper scripts.

This creates a severe systems mismatch: the embedding and reranking bottleneck.

The Naive Embedding and Reranking Serving Breakdown: ┌─────────────────────────────────────────────────────────────────┐ │ Bottleneck 1: Rectangular Batch Padding Waste │ │ • Sentences in a batch vary from 20 to 512 tokens │ │ • Naive collators pad all sequences to the max length with 0s │ │ ❌ Over 65% of GPU Tensor Core cycles compute empty <pad> ops! │ │ ❌ VRAM memory consumption explodes on long documents! │ ├─────────────────────────────────────────────────────────────────┤ │ Bottleneck 2: The Cross-Encoder Quadratic Latency Wall │ │ • Cross-encoders concatenate Query + Passage into self-attention│ │ • Reranking 100 candidate chunks requires 100 forward passes │ │ ❌ Unbatched PyTorch inference takes 300ms–800ms! │ │ ❌ Stalls real-time conversational agents before LLM starts! │ ├─────────────────────────────────────────────────────────────────┤ │ Bottleneck 3: Python GIL Locks & Uncoalesced Requests │ │ • Requests arriving 5ms apart execute sequentially │ │ ❌ Python GIL serializes tensor allocations │ │ ❌ Hardware sits 80% idle while request queues back up │ └─────────────────────────────────────────────────────────────────┘ Result: Slow retrieval, wasted cloud GPU expenditure, and latency-choked RAG pipelines!

1. The Compute Waste of Rectangular Batch Padding

In real-world data extraction and document indexing, incoming text sequences vary widely in length:

  • A user query might contain 12 tokens, a product title contains 25 tokens, while a scraped article section spans 480 tokens.
  • Traditional PyTorch batching requires assembling rectangular 2D tensors of shape (batch_size, max_seq_len). Shorter sequences are padded with zero tokens (<pad>).
  • Because self-attention complexity scales quadratically with sequence length ($\mathcal{O}(N^2)$), over 60% to 75% of GPU matrix multiplications are performed on padding tokens that are subsequently discarded during mean or CLS pooling.

2. The Cross-Encoder Latency Wall in Two-Stage RAG

Dense bi-encoder embedding models (such as BGE, E5, or ModernBERT) map queries and documents into separate vector spaces. While fast, bi-encoders frequently retrieve false-positive candidates due to lexical ambiguity.

To guarantee high precision, modern RAG systems deploy cross-encoder rerankers (such as BGE-Reranker-Large or Cohere Rerank) to rescore candidate chunks by jointly attending over [CLS] Query [SEP] Passage [SEP]:

  • Rescoring 50 to 100 candidate passages requires evaluating millions of cross-attention weights.
  • When served via standard Hugging Face pipelines, cross-encoder scoring takes 350ms to 900ms, introducing unacceptable lag into real-time voice and search workflows.

3. The Concurrency Breakdown of Generic Web Wrappers

Serving machine learning models behind general-purpose web frameworks (such as raw Flask or FastAPI) fails under concurrent production traffic:

  • Requests arriving asynchronously are processed one by one or queued statically.
  • The runtime lacks continuous dynamic batching to merge requests that arrive within a 2ms window into a single fused GPU tensor pass.
  • High-end GPUs (like NVIDIA A100, H100, or RTX 4090) operate at less than 15% compute utilization, while incoming API requests timeout under load.

The Solution: Infinity’s High-Throughput Serving Architecture

To eliminate the embedding and reranking bottleneck and bring vLLM-level serving efficiency to representation models, Michael Feil developed Infinity.

Infinity is an open-source, high-throughput serving engine optimized specifically for text embeddings, multi-modal vision-text encoders, and cross-encoder rerankers. Built on an asynchronous dynamic batching scheduler, FlashAttention-2 unpadding, and support for multiple compiled execution backends (PyTorch with Torch.compile, ONNX Runtime, and NVIDIA TensorRT), Infinity achieves sub-5ms latency and up to 10x higher throughput than standard serving setups.

Infinity High-Throughput Serving Topology: ┌─────────────────────────────────────────────────────────────────┐ │ Client Ingestion: Concurrent /v1/embeddings & /v1/rerank Calls │ └────────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Infinity Asynchronous Scheduler & Request Queue │ │ • Non-blocking asyncio event loop groups concurrent requests │ │ • Sequence Length Sorting: Groups similar token lengths together│ │ • Dynamic Batching: Merges up to 256 requests in a 2ms window │ └────────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FlashAttention-2 Unpadding & Zero-Padding Memory Core │ │ • Concatenates variable-length inputs into 1D continuous tensor │ │ • Passes cumulative sequence offsets (cu_seqlens) │ │ • Zero compute spent on <pad> tokens! Full Tensor Core saturation│ └────────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Multi-Engine Compiled Backends (PyTorch / ONNX / TensorRT) │ │ • PyTorch + torch.compile: Inductor fuses pooling & LayerNorm │ │ • ONNX Runtime / TensorRT: FP16 & INT8 graph kernel optimization│ │ • Sub-4.5ms Cross-Encoder Reranking for 100 candidate passages │ └────────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Production OpenAI & Cohere Compatible HTTP Endpoints │ │ • POST /v1/embeddings • POST /v1/rerank • POST /v1/classify │ └─────────────────────────────────────────────────────────────────┘

The Architectural Pillars of Infinity

  1. Dynamic Continuous Batching: Infinity collects incoming requests in an asynchronous queue and dynamically coalesces them into batched inference passes based on max latency and batch size thresholds. Requests arriving while a batch is executing are scheduled immediately for the subsequent iteration, maximizing GPU occupancy without adding perceptible tail latency.
  2. FlashAttention-2 Unpadding (Zero-Padding Computation): Rather than padding short sentences with zeros, Infinity strips padding tokens completely. It flattens the batch of variable-length sequences into a single 1D tensor of active tokens alongside a cumulative sequence length vector (cu_seqlens). FlashAttention-2 computes exact self-attention strictly across valid tokens, slashing memory bandwidth overhead by up to 70%.
  3. Sequence Length Sorting: Before building an unpadded batch, Infinity’s scheduler sorts candidate texts by sequence length. Grouping sequences of similar lengths together minimizes variance in kernel workgroup execution, preventing thread divergence and maximizing SM (Streaming Multiprocessor) efficiency.
  4. Multi-Engine Execution Providers: Infinity decouples the model architecture from the execution runtime. Depending on the target hardware, developers can run:
    • PyTorch Engine (--engine torch): Supports FlashAttention-2 and dynamic shapes with torch.compile.
    • Optimum / ONNX Runtime (--engine optimum): Highly optimized graph execution on both NVIDIA GPUs and Intel/AMD CPUs via OpenVINO and oneDNN.
    • TensorRT Engine (--engine tensorrt): Fully compiled TensorRT plans achieving the theoretical maximum FLOPS on NVIDIA datacenter silicon.
  5. Dual Capability: Embeddings + Cross-Encoder Rerankers: Unlike engines that only handle embedding models, Infinity natively serves both bi-encoder embeddings and cross-encoder rerankers within the same daemon instance, exposing standardized OpenAI /v1/embeddings and Cohere-compatible /v1/rerank REST interfaces.

Comparing Embedding and Reranker Serving Engines

The table below contrasts standard Hugging Face serving, Hugging Face Text Embeddings Inference (TEI), and Infinity:

Embedding Engine Architecture Comparison: ┌───────────────────────────┬───────────────────────────────┬───────────────────────────────┬───────────────────────────────┐ │ Metric / Feature │ Hugging Face Pipelines / Flask│ Text Embeddings Inference(TEI)│ Infinity Serving Engine │ ├───────────────────────────┼───────────────────────────────┼───────────────────────────────┼───────────────────────────────┤ │ Batching Mechanism │ Static / Sequential │ Dynamic Rust Batching │ Asynchronous Dynamic Batching │ │ Padding Handling │ Rectangular Zero-Padding (OOM)│ Unpadding (Candle / FlashAttn)│ FlashAttention-2 Unpadding │ │ Sequence Sorting │ ❌ None │ ⚠️ Basic length bucketing │ ✅ Optimized Sequence Sorting │ │ Cross-Encoder Reranking │ ⚠️ Slow Python loop │ ✅ Supported (Candle) │ ✅ Sub-5ms Cross-Encoder Engine│ │ Execution Backends │ PyTorch Eager │ Rust Candle / PyTorch C++ │ PyTorch, ONNX, and TensorRT │ │ Model Support │ All Hugging Face Models │ Subset of architectures │ BGE, ModernBERT, E5, CLIP, etc│ │ API Standards │ Custom endpoints │ Custom / Partial OpenAI │ Full OpenAI & Cohere REST API │ │ CPU Acceleration │ Basic PyTorch │ Basic AVX │ Intel OpenVINO / oneDNN │ └───────────────────────────┴───────────────────────────────┴───────────────────────────────┴───────────────────────────────┘

Deep Dive into Infinity Core Subsystems & Execution Lifecycle

To understand how Infinity achieves high throughput and sub-5ms reranking speeds, let’s explore its core architectural components: The Dynamic Batching Scheduler, FlashAttention-2 Unpadding, and the Cross-Encoder Optimization Pipeline.

Infinity Internal Engine: Infinity Engine ├──► 1. Async Scheduler: Ingests requests via non-blocking asyncio queue with time-slicing ├──► 2. Sequence Sorter: Groups texts by token length to eliminate SM thread divergence ├──► 3. FA2 Unpadding: Flattens batches to 1D token streams, passing cu_seqlens offsets ├──► 4. Cross-Encoder Worker: Fused query-passage scoring with half-precision fp16/bf16 └──► 5. REST Dispatcher: Emits standard OpenAI /v1/embeddings and Cohere /v1/rerank JSON

1. The Dynamic Batching Scheduler and Sequence Sorter

When hundreds of concurrent requests arrive at Infinity’s HTTP server:

  • An asynchronous queue (asyncio.Queue) captures incoming payload items without blocking the main event loop.
  • The scheduler checks two configurable thresholds: batch_delay (the maximum time to wait for incoming items, typically 0.5ms–2.0ms) and max_batch_size (e.g., 64–256 sequences).
  • Once triggered, the scheduler inspects the token lengths of all queued items.
  • It applies sequence length sorting, grouping items with comparable token lengths into the same execution batch. This avoids mixing a 15-token sentence with a 512-token document, minimizing the execution span of GPU thread blocks.

2. FlashAttention-2 Unpadding & Zero-Padding Computation

In standard attention, computing attention scores requires a 4D tensor $Q, K, V \in \mathbb{R}^{B \times H \times N \times D}$, where $N$ is the maximum sequence length in the batch.

Infinity implements FlashAttention-2 Unpadding:

  • Given $B$ sequences with variable token lengths $L_1, L_2, \dots, L_B$, Infinity flattens all non-padding tokens into a continuous 1D token sequence of length $N_{\text{total}} = \sum_{i=1}^B L_i$.
  • It creates a 1D cumulative sequence length tensor:

$$\mathbf{cu_seqlens} = [0, L_1, L_1 + L_2, \dots, N_{\text{total}}]$$

  • The FlashAttention-2 GPU kernel reads $\mathbf{cu_seqlens}$ directly. It restricts the self-attention softmax calculation strictly to the boundaries $[cu_seqlens[i], cu_seqlens[i+1])$ for sequence $i$.
  • Result: Padding tokens are never transferred from HBM to SRAM, and zero floating-point operations are performed on empty padding.

3. Cross-Encoder Reranking Engine Optimization

A cross-encoder model evaluates the joint semantic relevance between a single query $q$ and $M$ candidate passages $&#123;p_1, p_2, \dots, p_M&#125;$:

  • Standard implementations serialize $M$ separate transformer forward passes.
  • Infinity constructs a single unpadded batch containing all $M$ query-passage pairs:

$$\text{Pair}_i = [\text{CLS}] \circ q \circ [\text{SEP}] \circ p_i \circ [\text{SEP}]$$

  • The model executes a single forward pass through the transformer encoder in half-precision (fp16 or bf16).
  • A specialized pooling head extracts the [CLS] token activation and projects it through a linear layer to produce logit scores $\mathbf{s} \in \mathbb{R}^M$.
  • Softmax or sigmoid normalization transforms the logits into calibrated relevance probabilities in under 4.5 milliseconds for 100 documents.

Hands-On Implementation: Deploying Infinity for Embeddings and Reranking

Let’s walk through installing Infinity, launching dedicated embedding and reranking servers, and querying both endpoints concurrently in Python.

1. Environment Setup and Installation

Install infinity-emb with support for all GPU execution providers and FlashAttention:

# Create a fresh Python environment conda create -n infinity-env python=3.10 -y conda activate infinity-env # Install Infinity with PyTorch, ONNX, and Server extras pip install "infinity-emb[all]"

2. Launching an Infinity Dense Embedding Server via CLI

Launch a high-throughput embedding server serving a state-of-the-art encoder (such as BAAI/bge-large-en-v1.5 or answerdotai/ModernBERT-base):

# Launch Infinity on port 7997 using PyTorch + FlashAttention-2 infinity_emb v2 \ --model-id-or-path BAAI/bge-large-en-v1.5 \ --port 7997 \ --host 0.0.0.0 \ --batch-size 64 \ --engine torch \ --dtype float16

The server initializes the model, verifies FlashAttention-2 kernels, and exposes /v1/embeddings at http://localhost:7997.

3. Serving a Cross-Encoder Reranker with Sub-5ms Latency

In a separate terminal or Docker container, launch an Infinity instance dedicated to cross-encoder reranking:

# Launch Cross-Encoder Reranker on port 7998 using TensorRT/ONNX infinity_emb v2 \ --model-id-or-path BAAI/bge-reranker-large \ --port 7998 \ --host 0.0.0.0 \ --batch-size 128 \ --engine optimum \ --dtype float16

4. Querying Embedding and Reranking Endpoints in Python

Query both endpoints concurrently using standard Python httpx and openai clients:

import time import httpx from openai import OpenAI # 1. Connect to Infinity Embedding Endpoint via OpenAI SDK embed_client = OpenAI( base_url="http://localhost:7997/v1", api_key="not-needed" ) # Generate dense vector embeddings for documents documents = [ "aiFetchly provides autonomous web data extraction and desktop AI copilots.", "PostgreSQL pgvector enables native HNSW indexing inside relational tables.", "Infinity is a high-throughput serving engine for embeddings and rerankers.", "Apple MLX optimizes machine learning on Apple Silicon unified memory." ] print("--- Generating Dense Embeddings ---") start_time = time.time() embed_response = embed_client.embeddings.create( model="BAAI/bge-large-en-v1.5", input=documents ) embed_latency = (time.time() - start_time) * 1000 print(f"Generated {len(embed_response.data)} embeddings in {embed_latency:.2f} ms") print(f"Embedding Vector Dimension: {len(embed_response.data[0].embedding)}\n") # 2. Query Infinity Cross-Encoder Rerank Endpoint rerank_url = "http://localhost:7998/v1/rerank" query = "How do I serve embeddings with high throughput and low latency?" rerank_payload = { "model": "BAAI/bge-reranker-large", "query": query, "documents": documents, "top_n": 2, "return_documents": True } print("--- Executing Cross-Encoder Reranking ---") start_time = time.time() with httpx.Client() as http_client: rerank_response = http_client.post(rerank_url, json=rerank_payload, timeout=10.0) rerank_data = rerank_response.json() rerank_latency = (time.time() - start_time) * 1000 print(f"Reranked {len(documents)} documents in {rerank_latency:.2f} ms") for rank in rerank_data["results"]: idx = rank["index"] score = rank["relevance_score"] doc_text = rank["document"]["text"] print(f"Rank {idx+1} [Score: {score:.4f}]: {doc_text}")

Production Architecture & High-Scale RAG Integration for aiFetchly

In automated data extraction, competitor price monitoring, and desktop developer copilots powered by AiFetchly, deploying Infinity forms the high-speed retrieval engine behind enterprise knowledge indexing.

aiFetchly Two-Stage RAG Pipeline with Infinity: ┌─────────────────────────────────────────────────────────────────┐ │ aiFetchly Web Scraper & Knowledge Ingestion Engine │ └────────────────────────────────┬────────────────────────────────┘ │ (Streams Thousands of Scraped Articles & PDF Text Chunks) ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Infinity Embedding Cluster (Port 7997: bge-large-en-v1.5) │ │ • Slashes batch padding using FlashAttention-2 unpadding │ │ • Indexes 25,000+ tokens/sec directly into Vector Storage │ └────────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ In-Process Vector Store (ChromaDB / LanceDB) │ │ • Stage 1: Sub-5ms ANN Search retrieves top 50 candidates │ └────────────────────────────────┬────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Infinity Cross-Encoder Cluster (Port 7998: bge-reranker-large) │ │ • Stage 2: Cross-attends Query + 50 candidate passages │ │ • Rescores and filters to Top 5 verified chunks in < 6ms │ └───────────────────┬─────────────────────────────────┬───────────┘ │ │ (Passes High-Precision Context) (Orchestrates Multi-Model Routing) ▼ ▼ ┌───────────────────────────────────────┐ ┌───────────────────────────────┐ │ RouteLLM Dynamic Dispatcher │ │ FastMCP & LangGraph Agent │ │ • Routes routine tasks to local MLX │ │ • Standardized tool execution │ │ • Routes complex logic to Claude 3.5 │ │ • Cyclical state validation │ └───────────────────────────────────────┘ └───────────────────────────────┘

1. High-Volume Scraped Content Indexing

When AiFetchly crawls e-commerce stores, technical documentation, or financial filings:

  • Tens of thousands of text chunks must be embedded into vector space in parallel.
  • By deploying Infinity with dynamic batching, aiFetchly vectorizes scraped content at over 25,000 tokens per second per GPU, preventing ingestion queues from backing up.
  • Dense embeddings are committed directly into embedded ChromaDB or LanceDB collections with zero network overhead.

2. Two-Stage Retrieval with Cross-Encoder Precision

To eliminate hallucinations in mission-critical desktop intelligence:

  • Stage 1 (High-Recall Candidate Retrieval): Approximate Nearest Neighbor (ANN) search retrieves the top 50 candidate chunks from vector storage in sub-5ms.
  • Stage 2 (High-Precision Cross-Encoder Reranking): Infinity’s reranking server cross-examines the query against the 50 candidates, scoring them and isolating the top 5 verified chunks in under 6 milliseconds.
  • This two-stage pipeline eliminates false-positive context chunks before they ever reach the generative model.

3. Cost-Optimized Agent Orchestration with RouteLLM and FastMCP

Integrating Infinity into the broader agent ecosystem:

  • Cost-Optimized Generation with RouteLLM: Once Infinity retrieves and reranks verified context, the query is dispatched through RouteLLM, routing routine synthesis to local Apple MLX models or escalating to frontier cloud APIs.
  • Standardized MCP Tool Exposing via FastMCP: Expose Infinity’s /v1/rerank and /v1/embeddings APIs as standardized Model Context Protocol tools via FastMCP, allowing desktop agents to rescore documents dynamically.
  • Agent Workflow State Machines with LangGraph: Coordinate cyclical retrieval, fact verification, and synthesis nodes inside LangGraph.
  • Sandboxed Script Execution via E2B: Safely execute dynamic Python data processing scripts inside E2B Firecracker microVMs.
  • Continuous Retrieval Evaluation with DeepEval: Measure retrieval context recall, context precision, and latency percentiles using DeepEval and Arize Phoenix telemetry to ensure zero retrieval degradation over time.

Summary: Accelerating RAG Pipelines with Infinity

For too long, representation models in RAG architectures were treated as secondary citizens, trapped in slow, unbatched Python loops while generative LLMs received all serving optimizations.

Infinity bridges this gap by bringing production-grade serving infrastructure to vector embeddings and cross-encoder rerankers. Combining asynchronous dynamic batching, FlashAttention-2 unpadding, sequence sorting, and multi-engine compilation across PyTorch, ONNX, and TensorRT, Infinity eliminates padding waste and delivers sub-5ms retrieval latency. In automated web intelligence, large-scale knowledge indexing, and desktop AI copilots engineered with AiFetchly, Infinity provides the high-throughput retrieval foundation required for enterprise-grade generative intelligence.