llm-optimizationsemantic-cachingvector-databaseredistutorial

Semantic Caching for LLMs: Vector Similarity Caching Guide

September 19, 2026

Semantic Caching Architecture: Query Embedding, Vector Similarity Lookup in Redis, and Instant Completion Return

The Problem: The High Cost and Latency of Repetitive LLM Queries

As generative AI applications scale from internal prototypes to millions of active users, engineering teams encounter two brutal economic realities: token inference costs and generation latency.

In production conversational agents, enterprise search portals, and customer support bots, users rarely ask completely novel questions. A significant percentage of incoming traffic consists of semantic variations of previously answered inquiries:

“How do I cancel my subscription?”
“Where can I cancel my paid membership plan?”
“Steps to terminate my monthly subscription.”

Why Traditional Web Caches Fail

In traditional web infrastructure, caching relies on deterministic key-value stores like Redis or Memcached using exact string hashes (e.g., SHA-256(prompt)). If a user modifies capitalization, alters punctuation, or swaps a synonym, the SHA-256 hash changes completely.

Standard key-value caching achieves an abysmal <5% cache hit rate in conversational AI applications. Every minor linguistic variation forces the backend to dispatch a full, expensive request to frontier LLMs—waiting 2 to 5 seconds and paying full token input/output prices for an answer the system already computed moments earlier.

The Solution: The Semantic Caching Architectural Pattern

To bridge the gap between human language variation and cache efficiency, modern AI gateway architectures implement Semantic Caching.

Instead of matching identical strings, a semantic cache converts incoming queries into high-dimensional vector embeddings, searches an Approximate Nearest Neighbor (ANN) index in a vector database (such as Redis Vector Store, Qdrant, or Milvus), and checks if the cosine similarity to a previously answered question exceeds a strict similarity threshold.

Semantic Caching Pipeline: Exact Hash Check, Vector ANN Similarity Search, Threshold Guard, and Cache Population

Comparing Caching Paradigms in AI Systems

To understand where semantic caching fits in modern infrastructure, compare the three predominant caching layers:

Caching Mechanism	Location	Lookup Key	Cache Hit Latency	Cost Savings	Primary Limitation
Exact Hash (Redis KV)	Application Gateway	Exact String SHA-256	< 2ms	100% on hit	Zero tolerance for phrasing differences (<5% hit rate)
Model Prompt Caching	LLM Inference Engine	Common Prefix Tokens	300ms – 1,500ms	50% – 80% on inputs	Still incurs full output generation latency and tokens
Semantic Caching	Application Gateway	Dense Vector Embeddings	15ms – 40ms	100% on hit (Zero LLM cost)	Requires embedding generation and threshold tuning

Core Mechanics: Exact Fast Path, Embeddings, and Thresholds

A production-grade semantic cache deploys a dual-tier lookup strategy:

1. The Tier-1 Exact Hash Fast Path

Generating an embedding for an incoming prompt takes approximately 20ms to 50ms and costs a fraction of a cent. Before generating an embedding, the gateway computes an exact SHA-256 hash of the normalized prompt (lowercase, trimmed whitespace). If an identical string was submitted recently, the gateway returns the cached response in < 2ms with zero embedding cost.

2. The Tier-2 Semantic Vector Search

If the exact hash misses, the gateway generates a compact vector embedding of the user query (e.g., using text-embedding-3-small). It queries the vector database using cosine distance:

$$\text{Cosine Similarity} = \frac{\mathbf{u} \cdot \mathbf{v}}{|\mathbf{u}| |\mathbf{v}|}$$

If the nearest neighbor vector has a similarity score higher than the calibrated threshold (typically $0.92 \le \text{Similarity} \le 0.95$), the cache considers it a match and returns the cached answer.

3. Calibrating Distance Thresholds to Prevent False Positives

Threshold selection represents the critical trade-off in semantic caching:

Threshold Too Low (< 0.88): High cache hit rate, but risks catastrophic semantic mismatches (e.g., confusing “How do I upgrade my plan?” with “How do I cancel my plan?”).
Threshold Too High (> 0.97): Near-zero false positive rate, but misses natural synonyms and syntactic reorganizations, degrading the cache hit rate.
Production Sweet Spot (0.92 – 0.94): Safely absorbs synonyms, typos, and grammatical shifts while rejecting queries with opposing user intent.

Implementation: Building a Semantic Cache with Redis in TypeScript

Let us build a complete, production-ready semantic cache in TypeScript using the official Redis client (redis) and OpenAI embeddings.

Step 1: Define Schemas and Initialize Redis Vector Index

First, connect to Redis and define the RediSearch schema configured with Hierarchical Navigable Small World (HNSW) vector indexing:

import { createClient } from 'redis';

export interface CacheEntry {
  prompt: string;
  response: string;
  model: string;
  createdAt: number;
}

export const REDIS_INDEX_NAME = 'idx:semantic_cache';
export const EMBEDDING_DIM = 1536; // Dimensions for text-embedding-3-small

export async function initializeCacheIndex(redis: ReturnType<typeof createClient>) {
  try {
    // Check if index already exists
    await redis.ft.info(REDIS_INDEX_NAME);
    console.log('[Cache] Semantic index already exists.');
  } catch {
    console.log('[Cache] Creating RediSearch HNSW vector index...');
    await redis.ft.create(
      REDIS_INDEX_NAME,
      {
        '$.prompt': {
          type: 'TEXT',
          AS: 'prompt',
        },
        '$.response': {
          type: 'TEXT',
          AS: 'response',
        },
        '$.embedding': {
          type: 'VECTOR',
          ALGORITHM: 'HNSW',
          TYPE: 'FLOAT32',
          DIM: EMBEDDING_DIM,
          DISTANCE_METRIC: 'COSINE',
          AS: 'vector',
        },
      },
      {
        ON: 'JSON',
        PREFIX: 'cache:query:',
      },
    );
    console.log('[Cache] Vector index successfully registered.');
  }
}

Step 2: Implement Embedding Generation and Vector Querying

Convert text queries to binary vector buffers and perform radius searches against Redis:

import OpenAI from 'openai';

const openai = new OpenAI();

export async function getQueryEmbedding(text: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: text.trim().toLowerCase(),
  });
  return response.data[0].embedding;
}

// Convert float32 array to binary buffer required by Redis HNSW
export function float32ToBuffer(floats: number[]): Buffer {
  return Buffer.from(new Float32Array(floats).buffer);
}

Step 3: Implement the Dual-Tier Semantic Cache Interceptor

Orchestrate the exact-match fast path, vector similarity fallback, and automated cache population:

import crypto from 'crypto';
import { type createClient } from 'redis';
import { getQueryEmbedding, float32ToBuffer, REDIS_INDEX_NAME } from './cache-utils';

export class SemanticCache {
  private redis: ReturnType<typeof createClient>;
  private similarityThreshold: number; // e.g., 0.92
  private defaultTTLSeconds: number;

  constructor(
    redisClient: ReturnType<typeof createClient>,
    similarityThreshold = 0.92,
    defaultTTLSeconds = 86400, // 24 hours
  ) {
    this.redis = redisClient;
    this.similarityThreshold = similarityThreshold;
    this.defaultTTLSeconds = defaultTTLSeconds;
  }

  public async query(prompt: string): Promise<string | null> {
    const normalizedPrompt = prompt.trim().toLowerCase();

    // 1. TIER-1 FAST PATH: Exact SHA-256 Hash Lookup (<2ms)
    const exactHash = crypto.createHash('sha256').update(normalizedPrompt).digest('hex');
    const exactHit = await this.redis.get(`cache:exact:${exactHash}`);
    if (exactHit) {
      console.log('[Cache Hit] Tier-1 Exact Match returned.');
      return exactHit;
    }

    // 2. TIER-2 SLOW PATH: Vector Embedding & Similarity Search (~30ms)
    const queryVector = await getQueryEmbedding(normalizedPrompt);
    const vectorBuffer = float32ToBuffer(queryVector);

    // Redis Vector Search query using KNN (k=1)
    const searchQuery = `*=>[KNN 1 @vector $BLOB AS score]`;
    const searchResults = await this.redis.ft.search(REDIS_INDEX_NAME, searchQuery, {
      PARAMS: { BLOB: vectorBuffer },
      RETURN: ['score', 'response', 'prompt'],
      DIALECT: 2,
    });

    if (searchResults.total > 0) {
      const topMatch = searchResults.documents[0].value;
      const cosineDistance = parseFloat(topMatch.score as string);
      const cosineSimilarity = 1 - cosineDistance;

      if (cosineSimilarity >= this.similarityThreshold) {
        console.log(
          `[Cache Hit] Tier-2 Semantic Match! Score: ${cosineSimilarity.toFixed(4)} | Match: "${topMatch.prompt}"`,
        );
        return topMatch.response as string;
      }
    }

    console.log('[Cache Miss] No semantically similar prompt found above threshold.');
    return null;
  }

  public async set(prompt: string, response: string, model: string): Promise<void> {
    const normalizedPrompt = prompt.trim().toLowerCase();
    const exactHash = crypto.createHash('sha256').update(normalizedPrompt).digest('hex');

    // Populate exact cache key
    await this.redis.set(`cache:exact:${exactHash}`, response, {
      EX: this.defaultTTLSeconds,
    });

    // Generate vector and store in Redis JSON vector index
    const embedding = await getQueryEmbedding(normalizedPrompt);
    const key = `cache:query:${exactHash}`;

    await this.redis.json.set(key, '$', {
      prompt: normalizedPrompt,
      response,
      model,
      embedding,
      createdAt: Date.now(),
    });

    await this.redis.expire(key, this.defaultTTLSeconds);
    console.log(`[Cache Stored] Successfully indexed semantic query: "${prompt}"`);
  }
}

Cache Invalidation and Dynamic TTL Strategies

Deploying semantic caching into mission-critical systems requires defensive operational policies:

Namespace by System Prompt Hash: If you update your agent’s system prompt or underlying foundation model, existing cached completions become stale. Namespace your cache keys by the SHA-256 hash of the system prompt (e.g., cache:exact:${systemHash}:${userHash}) so prompt adjustments instantly invalidate legacy answers.
Dynamic TTL Based on Domain Volatility: Static facts (e.g., “How do I format an ISO date?”) can safely live in cache for 30 days. Volatile business data (e.g., “What is our current product pricing?”) should have shorter TTLs (e.g., 2 to 4 hours) or event-driven invalidation hooks triggered by CMS updates.
Bypass for PII and Sensitive Queries: Never cache prompts containing sensitive credentials, authenticated user IDs, or personal identifying information. Implement a pre-cache filter that flags PII patterns and forces a direct passthrough to the model.

Conclusion

Semantic caching bridges the gap between the flexibility of natural language and the performance requirements of production software. By serving up to 70% of repetitive requests through in-memory vector similarity searches in tens of milliseconds, engineering teams dramatically reduce cloud API expenditures while delivering instant response times to their users.