How Hybrid Search Improves Retrieval for RAG Systems

A practical hybrid search pipeline does five things. It applies access and version filters, retrieves candidates with a lexical method such as BM25, retrieves another set with dense embeddings, combines the two ranked lists, and optionally reranks the best candidates with a more expensive model. The resulting passages become the evidence available to the RAG system.

The point is not to make two search engines agree. It is to let each catch what the other misses. Lexical retrieval is strong when the query contains an exact policy number, product code, person, acronym, error message, or unusual phrase. Dense retrieval can connect a question to a passage that expresses the same idea in different words. A good fusion step rewards evidence found by either path and gives extra confidence to evidence found by both.

Start with one sparse index, one dense index, and reciprocal rank fusion. Keep the candidates, component ranks, filters, and final evidence visible in traces. Add a reranker only after measuring what the first stage retrieves. Hybrid search cannot rescue missing documents, broken parsing, bad chunk boundaries, or permissions applied after retrieval.

Lexical search scores overlap between query terms and document terms, with useful corrections for term rarity and document length. That makes it predictable around exact language. Search for an endorsement code such as IMT-23, and the code itself is often the strongest clue. Search for ‘when can an insurer recover a payment from another party?’, and the relevant document may use ‘subrogation’ without repeating the question's words.

Dense retrieval maps the query and text into vectors and ranks them by learned similarity. It can bridge that vocabulary gap, but a single vector must compress the meaning of a passage. Rare identifiers, negation, dates, and nearly identical clauses can become weak distinctions. The behaviour also depends on the embedding model's training data and the domain and language of the corpus.

The BEIR benchmark makes the broader lesson clear. Across 18 diverse retrieval datasets, BM25 remained a robust baseline, while dense and sparse neural systems varied out of domain. Reranking and late-interaction models were strong on average but required more computation. BEIR does not prove that one architecture will win on your internal knowledge base. It shows why a heterogeneous test set is more honest than a favourite demo query.

Hybrid search is therefore most valuable when real questions mix signals: codes and concepts, names and roles, quoted wording and paraphrase, or several languages. If a corpus is small, vocabulary is stable, and questions mostly use the same terms as the documents, a tuned lexical system may already be sufficient. Measure that baseline before adding a second index.

First, define the eligible corpus. Enforce tenant, role, retention, effective-date, and document-status constraints inside each retrieval path. Filtering an unauthorised result after it has been ranked or placed in a prompt is too late. Both retrievers should search the same allowed snapshot, even if their index implementations differ.

Second, retrieve more candidates than the generator will read. For example, each path might return its top candidates, but the final context may contain only a handful after fusion, deduplication, and reranking. The exact counts are tuning parameters, not universal defaults. Log how often the needed evidence is absent from both candidate lists; no downstream ranker can select a passage it never sees.

Third, collapse duplicate chunks and preserve source identity. Overlap, repeated headers, document versions, and boilerplate can otherwise occupy most of the fused list. Keep the document ID, section path, page or passage locator, effective date, and parent context beside every candidate. Retrieval is not finished when a chunk has a high score; a reviewer still needs to understand where it came from.

Finally, assemble context deliberately. Prefer a set that covers the question over five neighbouring copies of the same paragraph. Where a precise child chunk matches, recover the containing section if a definition, exception, table header, or earlier decision is needed. Hybrid retrieval finds candidates; context assembly decides what the model is actually allowed to read.

A BM25 score and a vector-similarity score do not share a natural scale. Adding them directly gives meaning to numbers that were produced by different models and can shift across queries, indexes, and implementations. Weighted score fusion can work, but it needs calibration on representative relevance judgements and continued monitoring when the corpus changes.

Reciprocal Rank Fusion avoids that score-comparability problem. For each document, it sums a contribution based on the document's position in every ranked list. A high rank contributes more; appearing in both lists helps. The original 2009 paper used a simple formula and found the unsupervised method competitive across its TREC and LETOR experiments.

RRF is a strong starting point, not a law. Its constant and candidate depth affect how quickly lower ranks lose influence. It cannot tell whether two retrieved chunks contradict each other, whether the newest version should supersede an older one, or whether one retriever is poor for a particular query type. Those decisions belong in metadata rules, evaluation slices, or a trained reranker.

Keep the fusion trace. For every final passage, record its lexical rank, dense rank, fusion score, and any reranker score. When an answer cites the wrong version, you should be able to see whether the failure began with eligibility, candidate retrieval, fusion, reranking, or context assembly. One opaque relevance number removes exactly the evidence needed to improve the system.

A cross-encoder reranker reads the query and each candidate together, allowing richer relevance judgements than a first-stage vector comparison. It is useful when the broad retrievers usually achieve good recall but order close candidates poorly. It adds latency and computation, so apply it to a bounded list and compare the gain against a fusion-only baseline.

Hybrid does not have to mean classic BM25 plus one embedding vector. SPLADE v2 learns sparse term representations that retain exact-match and inverted-index properties while expanding queries and documents into related vocabulary. It can bridge some semantic gaps inside a sparse retrieval system, although it adds model and indexing complexity beyond ordinary lexical search.

ColBERTv2 represents text at token granularity and uses late interaction between query and document vectors. That preserves finer matching signals than a single vector. Its authors also introduced residual compression to reduce the storage cost associated with multi-vector retrieval. The approach can be an effective retrieval or reranking layer, but its index and serving profile differ from a simple dense store.

These models are alternatives in a design space, not automatic upgrades. A learned sparse retriever may reduce the gap between lexical and semantic search. A late-interaction model may improve ranking without a separate cross-encoder. Each adds operational choices, and each can fail outside the distribution represented in its training and evaluation. Keep the simplest architecture that passes the real retrieval tests.

Multilingual knowledge work exposes weaknesses that an English-only average can hide. A user may ask in Hindi or Bengali while the governing document is in English. Product names may remain untranslated. Transliteration creates another form of mismatch. Evaluate each important language direction separately, including mixed-language queries and exact identifiers.

The MIRACL dataset was created to evaluate retrieval across 18 languages, which is a useful reminder that multilingual performance needs its own evidence. A model label that says ‘multilingual’ is not enough. Build relevance judgements from the languages, scripts, and document types that people actually use.

Some newer retrievers combine several modes. M3-Embedding, for example, was designed to support dense, sparse, and multi-vector retrieval across more than 100 working languages and varying text lengths. That makes unified systems possible, but the paper's benchmark results are not a substitute for testing your terminology, OCR quality, and cross-language questions.

Metadata deserves equal attention. Effective date, jurisdiction, policy version, source authority, and relationship to a parent case can matter more than textual similarity. Do not ask a reranker to infer facts that the ingestion pipeline could store explicitly. In connected knowledge work, retrieval quality comes from the text, its relationships, and the rules that determine which evidence is valid now.

Create a labelled set of real questions before tuning weights or adding models. Include exact-code lookups, paraphrases, rare entities, ambiguous acronyms, version-sensitive questions, multilingual queries, multi-document synthesis, and questions with no supported answer. Mark the passages required for each answer and the metadata constraints that make them eligible.

Measure candidate recall for lexical, dense, and unioned results. Then measure ranking quality with a metric such as reciprocal rank or nDCG, useful evidence in the final context, duplicate concentration, citation correctness, latency, and cost. Report results by query slice. A higher overall score can conceal a serious drop on renewals, tables, or one language.

Run ablations. Compare lexical only, dense only, fusion, and fusion plus reranking under the same corpus snapshot and context budget. Inspect disagreements rather than celebrating the average. Queries won only by lexical search often expose identifiers or domain language; queries won only by dense search reveal paraphrase or vocabulary gaps. Cases lost by both usually point to parsing, chunking, missing content, filters, or inadequate representations.

Stop adding layers when the additional complexity does not solve a measured failure. Hybrid search is unnecessary if one retriever meets the requirement, and it is insufficient if the answer depends on relationships that are absent from the indexed text. A knowledge graph, parent-child links, or explicit workflow state may be the missing retrieval surface.

The goal is not a more elaborate search diagram. It is a source-grounded path from question to admissible evidence that a person can inspect. If you have a real knowledge-work query where exact terms and conceptual matches keep pulling in different directions, share the query, source types, and failed evidence path at sahil@granveo.com.