Query Rewriting & HyDE: Hypothetical Document Embeddings to Boost Retrieval Relevance
In an idealized information retrieval system, the query and the target document inhabit the exact same linguistic domain. In production reality, however, they represent two radically asymmetric communication modalities: a user query is typically short, ambiguous, interrogative, and sparse ($<10$ tokens: e.g. "ac voltage drop formula 3 phase"), while the true answer resides in a verbose, dense, declarative engineering handbook ($>500$ tokens containing electrical derivations, table references, and mathematical proofs).
When a dense embedding model encodes these two texts into latent vector space, the geometric distance between them is distorted by their stark syntactic asymmetry. Hypothetical Document Embeddings (HyDE) (Gao et al., Carnegie Mellon University & Microsoft, 2022) introduces an ingenious paradigm shift: instead of searching with the raw user query, an instruction-tuned LLM generates a hypothetical, pseudo-document that answers the prompt. By embedding this synthetic expository document rather than the raw question, HyDE projects the search vector directly into document-document space, bypassing the asymmetry bottleneck completely.
Core Engineering Thesis
Deploying HyDE with multi-sample centroid averaging ($\mathbf{v}_{ ext{HyDE}} = rac{1}{N}\sum ext{Encoder}( ilde{d}_i)$) improves zero-shot out-of-domain retrieval (BEIR) by +27.7% in NDCG@10 without requiring any labeled fine-tuning datasets.
1. The Mathematics of HyDE Projection
Let $q$ denote the user query and $\mathcal{D} = \{d_1, \dots, d_M\}$ represent the corpus of indexed documents. Traditional dense retrieval computes relevance via inner product:
$$S_{ ext{raw}}(q, d) = \langle ext{Encoder}_{ ext{query}}(q), ext{Encoder}_{ ext{doc}}(d) angle$$
Under the HyDE framework, the retrieval pipeline is reformulated into a two-step generative process:
Step 1: Zero-Shot Hypothetical Document Sampling
An instruction-following LLM (e.g. GPT-4o-mini, Mistral-7B, or Llama-3-8B) is prompted with the raw query to generate a plausible, authoritative-sounding document $ ilde{d}$:
$$ ilde{d} \sim P_{ ext{LLM}}(d \mid q, ext{instruction})$$
Step 2: Unsupervised Multi-Sample Centroid Pooling
To eliminate random hallucination artifacts, the system samples $N$ independent hypothetical documents $\{ ilde{d}_1, \dots, ilde{d}_N\}$ and computes the mean embedding vector:
$$\mathbf{v}_{ ext{HyDE}} = rac{1}{N} \sum_{i=1}^N ext{Encoder}( ilde{d}_i)$$
The vector $\mathbf{v}_{ ext{HyDE}}$ is then submitted to the vector database to retrieve actual corpus documents using standard HNSW cosine search:
$$S_{ ext{HyDE}}(q, d) = \langle \mathbf{v}_{ ext{HyDE}}, ext{Encoder}(d) angle$$
Figure 1: HyDE operational flow: Transforming short user questions into synthetic expository documents to query vector indices in document-space.
2. Why False Details Do Not Corrupt Retrieval
A natural engineering objection to HyDE is: "If the LLM hallucinates incorrect facts in the hypothetical document, won't it retrieve false documents?"
Empirical analysis proves the contrary. In modern dense embedding spaces (e.g. OpenAI, BGE, Contriever), embeddings operate as a topological lossy compressor: they preserve semantic topic, genre, vocabulary distribution, and stylistic registers while abstracting away specific numerical values or proper nouns. For example, if a user queries a legal question about California employment law, and the LLM hallucinates an answer citing "California Labor Code Section 2802" instead of "Section 2804", the resulting vector still lands dead-center in the manifold of California statutory employee reimbursement provisions. The dense retriever uses this topical vector to pull the real, factually accurate statute from the database.
3. Beyond HyDE: Multi-Query Expansion & Sub-Query Decomposition
HyDE is most effective when paired with complementary query rewriting strategies:
| Query Rewriting Technique | Input Transformation | Optimal Production Use Case |
|---|---|---|
| HyDE (Hypothetical Documents) | Query $ o$ 150-word declarative pseudo-answer. | Abstract technical questions, conceptual lookups, and cold-start zero-shot domains. |
| Multi-Query Expansion | Query $ o$ 3–5 diverse synonymous variations. | Overcoming vocabulary mismatch in ambiguous or multi-disciplinary queries. |
| Step-Back Prompting | Query $ o$ High-level principle abstraction. | Complex scientific or architectural reasoning where foundational theory is needed. |
| Sub-Query Decomposition | Multi-part query $ o$ 3 independent sub-questions. | Comparative queries (e.g. "Compare Snowflake vs BigQuery cost models"). |
4. Empirical Retrieval Benchmarks (BEIR Benchmark)
| Dataset / Task | BM25 | Contriever (Dense Solo) | Contriever + HyDE | Relative Lift |
|---|---|---|---|---|
| TREC-COVID (Biomedical) | 0.656 | 0.274 (Dense Failure) | 0.613 | +123.7% |
| NFCorpus (Nutrition Science) | 0.325 | 0.317 | 0.362 | +14.2% |
| FiQA (Financial QA) | 0.236 | 0.245 | 0.298 | +21.6% |
| SciFact (Scientific Fact Checking) | 0.665 | 0.649 | 0.691 | +6.5% |
| Macro Average NDCG@10 | 0.470 | 0.371 | 0.491 | +32.3% Macro Lift |
5. Production Implementation (Python)
The following self-contained module demonstrates a production HyDE Retrieval Engine with multi-sample document generation, centroid pooling, and hybrid query fusion.
# -*- coding: utf-8 -*-
from typing import List, Dict, Any, Optional
import numpy as np
class HyDERetrievalEngine:
def __init__(self, embedding_dim: int = 128, alpha_query_blend: float = 0.3):
self.embedding_dim = embedding_dim
self.alpha_query_blend = alpha_query_blend
self.doc_ids: List[str] = []
self.doc_texts: List[str] = []
self.doc_vectors: np.ndarray = None
def _mock_embed(self, text: str) -> np.ndarray:
np.random.seed(abs(hash(text)) % (2**32))
vec = np.random.randn(self.embedding_dim)
return vec / (np.linalg.norm(vec) or 1.0)
def _mock_generate_hypothetical_doc(self, query: str, sample_idx: int) -> str:
# In production, call fast model (e.g. gpt-4o-mini or mistral-7b)
return (
f"Technical document addressing: {query}. In standard enterprise systems, "
f"the operating parameters involve verified configurations, specification constants, "
f"and established architectural protocols (sample {sample_idx})."
)
def index_documents(self, doc_ids: List[str], texts: List[str]):
self.doc_ids = doc_ids
self.doc_texts = texts
vecs = [self._mock_embed(t) for t in texts]
self.doc_vectors = np.array(vecs)
def retrieve_with_hyde(
self,
query: str,
num_hypothetical_samples: int = 3,
top_k: int = 5
) -> List[Dict[str, Any]]:
# 1. Embed raw user query
raw_query_vector = self._mock_embed(query)
# 2. Generate and embed N hypothetical documents
hypo_vectors = []
hypo_texts = []
for i in range(num_hypothetical_samples):
h_text = self._mock_generate_hypothetical_doc(query, i)
hypo_texts.append(h_text)
hypo_vectors.append(self._mock_embed(h_text))
# 3. Compute HyDE Centroid Vector
hyde_centroid = np.mean(hypo_vectors, axis=0)
hyde_centroid = hyde_centroid / (np.linalg.norm(hyde_centroid) or 1.0)
# 4. Blend HyDE Centroid with Raw Query Vector
fused_search_vector = (
(1.0 - self.alpha_query_blend) * hyde_centroid +
self.alpha_query_blend * raw_query_vector
)
fused_search_vector = fused_search_vector / (np.linalg.norm(fused_search_vector) or 1.0)
# 5. Search Vector Database
scores = np.dot(self.doc_vectors, fused_search_vector)
top_indices = np.argsort(-scores)[:top_k]
results = []
for idx in top_indices:
results.append({
"doc_id": self.doc_ids[idx],
"text": self.doc_texts[idx],
"score": round(float(scores[idx]), 4)
})
return results
6. Architectural Recommendations
- Use Fast, Cheap Models for Generation: Do not use frontier reasoning models ($15/M tokens) to generate hypothetical documents. Use fast sub-second models like Llama-3-8B, Gemini 2.0 Flash, or GPT-4o-mini with latency under 300ms.
- Blend HyDE with Raw Query Vectors ($lpha=0.3$): Pure HyDE can occasionally drift if the LLM's hallucination is completely unmoored. Blending 30% of the raw query embedding vector ensures exact-token anchors are never lost.
- Disable HyDE for Direct Entity Lookups: If a query contains strict identifiers (e.g.
INV-9281-XorRFC-8446), bypass HyDE and route directly to BM25 or standard dense search. HyDE is engineered for conceptual and descriptive queries.