Corrective RAG (CRAG): Evaluating Retrieved Document Quality Before Passing to Generation
In conventional Retrieval-Augmented Generation (RAG) architectures, the generation phase is completely subservient to the retrieval phase. Documents returned by vector similarity search are unconditionally concatenated into the prompt context window under the blind assumption that retrieval was successful. When the retriever fails—due to out-of-distribution queries, index sparsity, or ambiguous phrasing—the generation LLM is forced to synthesize an answer from irrelevant or contradictory context, leading to confident, catastrophic hallucinations.
Corrective Retrieval-Augmented Generation (CRAG) (Yan et al., 2024) fundamentally breaks this brittle open-loop dependency. By inserting an autonomous Retrieval Evaluator between the retrieval engine and the generation model, CRAG quantitatively assesses the quality of retrieved documents against the query. Depending on the evaluator's confidence, CRAG executes a Tri-State Decision Flow: refining correct documents into distilled knowledge strips, triggering an automated Web Search Fallback for incorrect documents, or combining internal and external streams for ambiguous queries.
Core Engineering Thesis
Deploying a lightweight Retrieval Evaluator with a tri-state threshold ($ au_U = 0.5, au_L = -0.5$) reduces LLM hallucination rates from 28.4% down to 9.2% (-67.6% reduction) and boosts PopQA retrieval-augmented accuracy by +18.6% on long-tail factual queries.
1. The Tri-State Decision Matrix Formulation
Let $Q$ be the user query and $\mathcal{D} = \{d_1, d_2, \dots, d_k\}$ be the set of documents retrieved by an upstream vector or hybrid search engine. The Retrieval Evaluator $E$ processes each document $d_i$ and assigns a continuous confidence score $s_i \in [-1, 1]$.
The system computes the aggregated confidence score $S = ext{Aggregate}(s_1, \dots, s_k)$ (typically the mean or maximum score of top candidates). Based on two predetermined hyperparameter thresholds—the Upper Threshold $ au_U$ and Lower Threshold $ au_L$—the system branches into one of three execution paths:
$$ ext{Action}(\mathcal{D}, Q) = egin{cases} \mathbf{Correct}, & ext{if } S \ge au_U \ \mathbf{Incorrect}, & ext{if } S \le au_L \ \mathbf{Ambiguous}, & ext{if } au_L < S < au_U \end{cases}$$
Figure 1: Corrective RAG (CRAG) decision state machine showing confident refinement, ambiguous fusion, and active web fallback.
2. Knowledge Strip Refinement Mechanics
Even when a document is verified as Correct, retaining the entire 500-token chunk introduces irrelevant fluff, introductory remarks, and tangent sentences. CRAG implements a fine-grained Knowledge Refinement Algorithm:
- Sentence Deconstruction: The document is partitioned into fine-grained atomic units called knowledge strips ($k_1, k_2, \dots, k_m$), each spanning 20 to 30 words.
- Strip-Level Scoring: The Evaluator scores each individual strip against query $Q$.
- Noise Purging: Strips scoring below a local relevance threshold $\gamma$ are discarded. The remaining high-salience strips are concatenated into a distilled, dense context payload.
This procedure eliminates up to 60% of irrelevant tokens from genuine documents, drastically sharpening LLM attention heads during generation.
3. Web Search Query Rewriting & Fallback
When the Evaluator diagnoses the retrieval state as Incorrect ($S \le au_L$), it detects that the internal vector database lacks coverage. Instead of allowing the generation model to hallucinate from irrelevant internal text, CRAG executes an automated fallback:
- Internal Context Purge: All internally retrieved chunks are purged from memory to prevent context contamination.
- Search Query Synthesis: An auxiliary instruction model rewrites the natural language query $Q$ into search-engine optimized keywords. For instance:
- User Query: "Why did company X reject the acquisition proposal in late 2024?"
- Search Rewrite:
"Company X" "acquisition proposal" reject reason 2024 regulatory
- Live Web Retrieval: The system queries external search APIs (e.g. Tavily, Bing Search API), extracts live web snippets, and filters them through the Knowledge Strip Refiner before passing to the final generation model.
4. Empirical Benchmarks (PopQA & PubHealth)
The efficacy of Corrective RAG is evaluated across standard open-domain and biomedical QA benchmarks:
| Architecture | PopQA (Accuracy) | PubHealth (Accuracy) | Hallucination Rate | Average Context Tokens |
|---|---|---|---|---|
| Direct LLM (No Retrieval) | 24.2% | 41.5% | 42.1% | 0 tokens |
| Standard RAG (Dense Only) | 44.2% | 68.1% | 28.4% | 1,840 tokens |
| Self-RAG | 54.9% | 73.4% | 16.8% | 1,420 tokens |
| Corrective RAG (CRAG) | 62.8% | 79.2% | 9.2% | 680 tokens |
Notice that CRAG achieves the highest factual accuracy while consuming only 680 context tokens (compared to 1,840 tokens for standard RAG). By distilling documents into knowledge strips and discarding false positives, CRAG simultaneously lowers latency, decreases API cost, and suppresses hallucinations.
5. Production Implementation (Python)
The following runnable module implements a complete Corrective RAG pipeline: a simulated Retrieval Evaluator, Knowledge Strip Decomposition, and dynamic fallback routing.
# -*- coding: utf-8 -*-
import re
from typing import List, Dict, Any, Tuple
class CorrectiveRAGEngine:
def __init__(self, tau_upper: float = 0.5, tau_lower: float = -0.5):
self.tau_upper = tau_upper
self.tau_lower = tau_lower
def _evaluator_score(self, query: str, document: str) -> float:
# Lightweight relevance evaluator (e.g., fine-tuned MiniLM or LLM classifier)
# Returns score in range [-1.0, 1.0]
q_tokens = set(re.findall(r'\w+', query.lower()))
d_tokens = set(re.findall(r'\w+', document.lower()))
intersection = q_tokens.intersection(d_tokens)
if not q_tokens:
return -1.0
overlap_ratio = len(intersection) / len(q_tokens)
# Scale [0.0, 1.0] to [-1.0, 1.0]
score = (overlap_ratio * 2.0) - 1.0
return max(-1.0, min(1.0, score))
def _decompose_knowledge_strips(
self,
document: str,
strip_len_words: int = 25
) -> List[str]:
# Deconstructs document into fine-grained atomic strips
words = document.split()
strips = []
for i in range(0, len(words), strip_len_words):
strip = " ".join(words[i : i + strip_len_words])
if len(strip.strip()) > 0:
strips.append(strip)
return strips
def _refine_knowledge(self, query: str, documents: List[str]) -> List[str]:
# Filters knowledge strips across documents, retaining only relevant tokens
retained_strips = []
for doc in documents:
strips = self._decompose_knowledge_strips(doc)
for strip in strips:
score = self._evaluator_score(query, strip)
if score >= -0.1: # Local strip filter threshold
retained_strips.append(strip)
return retained_strips
def _simulate_web_search(self, query: str) -> List[str]:
# Simulated external web search fallback
return [
f"[Live Web Result] Verified search data for query: '{query}' regarding authoritative enterprise sources.",
"[Live Web Result] Secondary corroborating source confirming external real-time data."
]
def execute_crag(
self,
query: str,
retrieved_docs: List[str]
) -> Dict[str, Any]:
if not retrieved_docs:
state = "Incorrect"
scores = []
aggregated_score = -1.0
else:
scores = [self._evaluator_score(query, doc) for doc in retrieved_docs]
aggregated_score = max(scores)
if aggregated_score >= self.tau_upper:
state = "Correct"
elif aggregated_score <= self.tau_lower:
state = "Incorrect"
else:
state = "Ambiguous"
final_context = []
actions_taken = [f"Evaluated state: {state} (Aggregated Score: {aggregated_score:.2f})"]
if state == "Correct":
refined = self._refine_knowledge(query, retrieved_docs)
final_context = refined
actions_taken.append(f"Refined internal documents into {len(refined)} high-salience knowledge strips.")
elif state == "Incorrect":
actions_taken.append("Internal documents rejected. Triggering external web search fallback.")
web_results = self._simulate_web_search(query)
final_context = web_results
elif state == "Ambiguous":
actions_taken.append("Ambiguous confidence. Executing hybrid internal refinement + web augmentation.")
internal_strips = self._refine_knowledge(query, retrieved_docs)
web_results = self._simulate_web_search(query)
final_context = internal_strips + web_results
return {
"query": query,
"decision_state": state,
"confidence_score": aggregated_score,
"context_payload": final_context,
"execution_log": actions_taken
}
6. Architectural Recommendations
- Train a 100M-Parameter Specialized Evaluator: Do not burn expensive frontier LLM tokens to evaluate document relevance. Fine-tune a lightweight model (e.g.,
RoBERTa-largeorbge-reranker-base) on binary relevant/irrelevant classification to execute evaluation in under 5ms. - Tune Thresholds Conservatively: In safety-critical applications (medical diagnosis, compliance auditing), raise $ au_U$ to 0.7 to ensure that only near-perfect internal matches bypass external verification.
- Cache Web Fallbacks: When a query triggers web search, immediately write the verified external results back to your internal vector database so future identical queries resolve as "Correct" on internal retrieval.