From keyword search to semantic discovery: how semantic search works

LLMs generate answers from their training data. They have no built-in access to your knowledge base, product catalog, or internal documents. To use them against your own data, you need a retrieval layer that fetches the right content at query time and passes it to the model as context. That retrieval layer needs to understand what the user is asking, not just which words they used. Keyword search matches tokens. Vector embeddings match meaning, and that distinction is what makes them the right foundation for AI-powered retrieval.

This article covers the retrieval stack: the mechanics of semantic search, how vector databases work, where pure embedding-based retrieval fails, and what it takes to build a system where retrieval quality holds up in production.

1. The limitations of traditional keyword search

How keyword-based search works

Keyword search operates on an inverted index: for every term in a corpus, the system records which documents contain it. At query time, it tokenizes the input and scores documents by how well their terms match the query.

The dominant scoring algorithm is BM25 (Best Matching 25). It weighs three factors: term frequency with diminishing returns for repetition, inverse document frequency where rare terms carry more signal, and length normalization that penalizes longer documents. The result is a ranked list where exact token matches drive relevance.

This works when users know the vocabulary the corpus uses. It breaks down when they do not.

Why keyword search struggles with meaning

Keyword systems have no representation of meaning, only identity. "Footwear for running" and "athletic shoes for jogging" share no tokens, so a keyword system scores them as unrelated. Synonyms, paraphrases, and natural language variation all become retrieval failures.

Ambiguity creates the opposite problem. "Python tutorial for data analysis" matches documents about the programming language and the reptile equally — the system has no way to tell them apart.

2. What semantic search is

Semantic search retrieves by meaning, not by token overlap. Where a keyword system asks "does this document contain these words?", a semantic system asks "does this document mean what this query means?"

The mechanism is vector embeddings: numerical representations of text that encode semantic relationships. Documents and queries with similar meanings produce vectors that land close together in a high-dimensional space — close enough that a similarity function can rank them by relevance without any token matching.

The pipeline has three stages:

• Encoding: An embedding model converts documents and queries into dense, fixed-dimension vectors.
• Indexing: Those vectors are stored in a vector database that supports approximate nearest neighbor (ANN) search.
• Retrieval: At query time, the query is encoded with the same model. The system returns the top-k vectors closest to it, typically scored by cosine similarity.

Retrieval quality depends on two choices above all others: the embedding model and the indexing configuration.

Types of semantic search

Semantic search requires two things: a way to represent meaning, and a similarity function for matching. Vector embeddings are the current industry standard for both, but they are one approach among several.

Vector embeddings convert text into dense numerical vectors using transformer-based models. A high-dimensional vector encodes semantic content across many linguistic axes simultaneously. Retrieval runs on cosine similarity or dot product. This is what most practitioners mean when they say "semantic search."

Hybrid search combines BM25 precision with semantic coverage. BM25 handles exact-term recall; the embedding model handles conceptual similarity; a cross-encoder reranker scores the merged candidate set for final ordering.

Lexical expansion maps query terms to semantically related terms before executing a keyword search. It addresses synonymy without vector infrastructure, at the cost of maintaining synonym dictionaries and missing paraphrases those dictionaries do not cover.

3. How vector databases work

A traditional database returns rows where a value exactly matches the query. A vector database works differently: it finds the vectors most similar to the query vector, even when there is no exact match. That approximation is intentional — you are retrieving by meaning, and meaning is rarely identical.

The pipeline has three stages:

• Indexing: Vectors are organized using an algorithm that trades a small amount of accuracy for query speed. HNSW (Hierarchical Navigable Small World) is a common production choice, offering strong recall at low latency. Other approaches, like IVFFlat (which clusters vectors using k-means before searching), PQ (Product Quantization), and LSH (Locality Sensitive Hashing), are used when memory is the primary constraint or the workload favors a different accuracy-speed tradeoff.
• Querying: The database compares the query vector against indexed vectors and returns the approximate nearest neighbors.
• Post-processing: Results are filtered by metadata, reranked, or deduplicated as needed.

A production vector database adds what any data system needs on top of raw similarity search: CRUD operations, metadata filtering, real-time updates without full reindexing, horizontal scaling, and access control.

Metadata filtering is worth highlighting specifically. Without it, retrieval has no way to scope results to a user, a department, a product category, or a permission level. That makes it a hard requirement for any multi-tenant or access-controlled deployment.

Choosing between the available options (Pinecone, Weaviate, Qdrant, pgvector, and Azure AI Search) depends on your infrastructure and deployment constraints.

4. The vector embedding pipeline in detail

Understanding the end-to-end pipeline clarifies where failures occur:

Step 1: Chunking

Long documents must be split into chunks before embedding. Chunk size determines what semantic unit gets a single vector. A chunk that is too small loses context; a chunk that is too large mixes topics and dilutes the embedding. Common strategies include fixed-size chunks with overlap, sentence-boundary splitting, and semantic chunking, which groups sentences by topic shift. Poor chunking is among the most frequent causes of retrieval failure in RAG systems.

Step 2: Embedding model

Each chunk is passed through an embedding model that produces a fixed-dimension vector. General-purpose models work well for broad content, while domain-specific models improve recall on specialized corpora in areas like legal, medical, or code.

Step 3: Index insertion

The vector, along with a reference to the original chunk and associated metadata such as document ID, author, timestamp, and access level, is written to the vector database.

Step 4: Query encoding

The user's query is passed through the same embedding model. The resulting vector is compared to stored vectors using the configured similarity metric.

Step 5: Retrieval and filtering

The database returns the top-k nearest vectors. Metadata filters can be applied pre- or post-retrieval to scope results. In RAG systems, the retrieved chunks are injected into the LLM's context window as grounding evidence before the model generates a response.

5. Hybrid search: why enterprises combine keyword and vector search

The strengths of lexical search

BM25 excels on exact terms. Part numbers, regulatory codes, SKUs, medication names, and legal citations all depend on the signal being in the exact string. It requires no GPU inference at query time, is predictable, and works well out of the box for structured data and known vocabulary.

The strengths of semantic search

Semantic retrieval handles natural language variation that keyword systems miss entirely. A user asking "what's the refund process?" retrieves documents about "return policies" and "money-back guarantees" without any synonym configuration.

It also transfers across languages. A multilingual embedding model can match a French query to an English document if the concepts align.

Why hybrid architectures often perform best

Dense retrieval works well for conceptual queries. It breaks down when specific technical language matters. A compliance officer searching "Article 17 GDPR right to erasure implementation" wants that exact regulatory reference, not documents that are conceptually adjacent to data privacy. The embedding model compresses the entire chunk into one vector, averaging out precise terminology in the process. BM25 does not have this problem.

The practical guidance: lean toward semantic for conceptual queries, lean toward BM25 for precise term lookups, and use metadata filtering to narrow the candidate set when the filter is highly selective.

Hybrid search in enterprise environments

For enterprise knowledge bases, the recommended default architecture is BM25 plus dense retrieval with RRF, followed by cross-encoder reranking on the top candidates.

Cross-encoders evaluate query and document together rather than independently, producing more accurate relevance scores at the cost of latency. Apply them after initial retrieval, not to the full corpus.

6. Real-world use cases for vector search

Enterprise knowledge management: Merck

Merck's domain experts were manually identifying chemical compounds from scientific literature to support sales decisions. Each review cycle took around six months.

Netguru built an AI R&D Assistant in five weeks. Experts upload PDFs; the system extracts compounds, assigns official identifiers, and cross-references Merck's product catalog. The core technical challenge was input size: scientific papers exceed LLM context limits, solved by splitting documents into chunks before processing.

The same task now completes in approximately six hours.

Customer support: heating systems manufacturer

A heating systems manufacturer ran customer support on manual processes, with information dispersed across systems and no 24/7 coverage.

Netguru built a RAG-based AI chatbot that retrieves content from the client's CMS in real time, handles multilingual queries, and manages conversation context across sessions. Content updates do not require redeployment.

The outcome: 24/7 support coverage, faster response times, and unified access to product information for customers and internal staff.

Chatguru: RAG-based retrieval in practice

Netguru's open-source RAG-based chatbot platform, Chatguru, is built around the same retrieval principles this article covers. It grounds responses in the client's own product catalog or knowledge base, retrieved at query time via semantic search. That architecture reduces the risk of the model generating answers that are plausible but not backed by actual inventory or documentation.

We used Chatguru to build a clinical knowledge assistant for the International Team for Implantology (ITI), a global association of over 25,000 dental professionals. Grounding responses exclusively in ITI's verified content significantly reduces hallucinations. The chatbot now answers clinical questions with 98% accuracy.

7. How to choose the right vector search setup

Selecting embedding models

The embedding model determines the quality ceiling of retrieval. General-purpose models are fast and broadly capable but underperform on specialized vocabularies. Domain-adapted models, fine-tuned on legal, medical, or code corpora, improve recall on those domains beyond what general-purpose models typically achieve.

Key dimensions to evaluate: embedding dimension, maximum input token length, and benchmark performance on datasets similar to your actual query distribution. For multilingual applications, use a multilingual model from the outset. Retrofitting later is expensive.

Choosing a vector database

The right choice depends primarily on your existing infrastructure:

• Azure stack: Azure AI Search supports hybrid search with semantic reranking natively.
• Full control required: Qdrant or Weaviate as self-hosted options with strong metadata filtering and hybrid support.
• Postgres-native: pgvector keeps retrieval inside an existing database, reducing operational overhead at the cost of ANN performance at scale.
• Minimal ops: Pinecone for managed infrastructure without index parameter tuning.

Cloud vs. self-hosted

Self-hosted gives you data sovereignty, no vendor lock-in, and cost control at scale, at the cost of operational capacity to manage upgrades and performance tuning. For teams handling sensitive data in health care, finance, or legal, self-hosting is often a compliance requirement rather than a preference. Chatguru runs as a Dockerized self-hosted deployment for exactly this reason: enterprise customers require that product catalogs and conversation data never leave their own infrastructure.

Cloud-managed services reduce time to production but shift costs to per-query or per-vector pricing, which scales with usage.

8. The future of semantic search

Multimodal embeddings

Google's Gemini Embedding 2, released in March 2026, is the first natively multimodal embedding model, mapping text, images, video, audio, and documents into a single embedding space. For media, product, and document-heavy applications this removes the need for separate retrieval pipelines per modality. A single index handles mixed-modal queries: "find products that look like this image and match this description."

Personalized semantic retrieval

Retrieval personalization adds a user-specific signal to the similarity computation. Click history, purchase history, and engagement patterns are embedded as a user vector and combined with the query vector at retrieval time. This shifts semantic search from "most relevant to this query" toward "most relevant to this user, for this query."

Agentic AI and retrieval

AI agents decompose multi-step goals into sub-queries, retrieve evidence for each, and synthesize results across multiple retrieval rounds. The retrieval layer becomes a first-class tool in the agent's action space, which places harder demands on latency and reliability than current conversational search requires.

A practical example of single-hop RAG is Chatguru, which grounds each answer in one semantic search pass over the business's own data rather than chaining multiple retrieval steps.

Real-time index freshness

Static indexes reflect the corpus at the time of last ingestion. As catalogs, knowledge bases, and regulatory documents update continuously, near-real-time index updates become a hard requirement. This is already a differentiating factor between vector database options: some require full reindexing to incorporate new data; others support streaming updates.

Conclusion

Semantic search is a pipeline. Chunking strategy, embedding model, indexing algorithm, filtering configuration, and evaluation setup each contribute to retrieval quality. Getting one right while ignoring the others produces a system that works in demos and fails in production.

The architecture that holds up in practice combines BM25 precision with semantic coverage. A reranker handles the cases neither method resolves cleanly. Retrieval quality gets measured continuously, not once at launch. That pattern applies to product discovery, internal knowledge assistants, and customer support systems alike.

The field is moving fast. Multimodal embeddings, personalized retrieval, and agentic multi-round retrieval are shifting from research to production.

FAQ

What is the difference between semantic search and keyword search?

Keyword search matches exact tokens between a query and an index. Semantic search converts both the query and documents into vector representations that encode meaning, then finds documents whose meaning is closest to the query, regardless of whether the same words appear.

What is a vector embedding in the context of search?

A vector embedding is a fixed-dimension numerical array that represents the semantic content of a text chunk. Chunks with similar meaning produce vectors that are geometrically close in the embedding space, which is what enables similarity-based retrieval.

What are the most common implementation mistakes?

In order of frequency: using a general embedding model on a specialized corpus, using fixed-size chunking without validating chunk coherence, skipping retrieval evaluation before launch, and ignoring metadata filtering that would have scoped retrieval correctly.

How is semantic search used in RAG systems?

In RAG (Retrieval-Augmented Generation), semantic search retrieves the document chunks most relevant to a user's query. Those chunks are injected into the LLM's context window as grounding evidence before the model generates a response. The quality of the generated answer depends directly on the quality of retrieval.