Hybrid Retrieval Strategies for RAG over Scientific Literature | Dr. Amit Puri | Dr. Amit Puri Foundation — Amplifying Human Ingenuity in the Age of AI

01. Executive Summary & Core Insight

Teams building retrieval-augmented generation (RAG) systems over scientific and technical literature must choose among sparse, dense, fusion, graph-augmented, and approximate-nearest-neighbor retrieval strategies — a choice that is frequently made by assumption rather than by measurement.

This empirical systems study by Dr. Amit Puri presents an empirical comparison of eighteen retrieval strategies (fifteen core architectures plus three factorial ablation configurations) evaluated on an expanded corpus of scientific PDF documents (44 arXiv-style papers, 1,709 pages, 9,558 structured chunks) across 22 curated benchmark queries (16 single-hop + 6 multi-hop reasoning) with explicit chunk-level ground truth and entity/relation annotations.

Seven Principal Empirical Findings

1. Lexical precision dominates on domain jargon: Pure BM25 (MRR = 0.551) outperforms all linear hybrid score blends (α ∈ {0.3, 0.5, 0.7}, MRR range 0.471–0.503), with retrieval quality degrading monotonically as dense score weight increases.
2. Rank-space fusion prevents score distortion: Reciprocal Rank Fusion (RRF, k=60) achieves MRR = 0.514 and Recall@1 = 0.318, substantially outperforming dense bi-encoders and mitigating the score-incompatibility penalty of linear combinations.
3. Multi-hop reasoning requires joint cross-attention: While cross-encoder reranking underperforms first-stage RRF on single-hop lexical queries due to web-passage training divergence (0.481 vs. 0.514 overall MRR), it establishes the benchmark ceiling on complex multi-hop reasoning queries (MRR = 0.408, NDCG@5 = 0.513, Recall@5 = 0.833 vs. RRF's 0.345 MRR).
4. Knowledge graph augmentation delivers structural coverage: Fusing an IDF-weighted NetworkX knowledge graph into RRF achieves high structural entity and relation coverage (EntCov = 0.902, RelCov = 0.727) and competitive ranking (MRR = 0.477, NDCG@5 = 0.528).
5. MMR diversification balances context quality: Post-deduplication Maximal Marginal Relevance (λ = 0.7) achieves the highest ranking quality with diversity (NDCG@5 = 0.547).
6. HNSW ANN vector retrieval is near-lossless at scale: Approximate Nearest Neighbor retrieval via Qdrant's HNSW index reproduces exact-search bi-encoder metrics identically (MRR = 0.318, NDCG@5 = 0.335), confirming sub-millisecond scalability without sacrificing fidelity.
7. Off-the-shelf dense bi-encoders diffuse coined technical vocabulary: Standard embeddings (all-MiniLM-L6-v2: 0.318 MRR) and standalone scientific bi-encoders (SPECTER2: 0.278 MRR) struggle on coined acronyms ("StarShell", "POMDP", "AgentRunner") without lexical grounding.

02. The Eight Research Questions (RQ1–RQ8)

To systematically evaluate retrieval and generation behavior, the study formulates eight falsifiable research questions spanning sparse vs. dense representations, fusion mathematics, reasoning topologies, and confidence estimation:

RQ1: Sparse vs. Dense Baselines

How do pure sparse (BM25), vector-space (TF-IDF), and pure neural dense bi-encoders (MiniLM, SPECTER2) compare on MRR, Recall@{1,3,5}, and NDCG@5?

RQ2: Linear Convex Score Fusion

Does linear score-level fusion improve retrieval quality over sparse retrieval alone at any evaluated α weighting (0.3, 0.5, 0.7)?

RQ3: Rank-Based Fusion & MMR

Does rank-based fusion (RRF) avoid score degradation, and how do Jaccard deduplication and MMR diversification alter ranking diversity?

RQ4: Cross-Encoder Division of Labor

How does cross-encoder reranking diverge between single-hop lexical queries and complex multi-hop reasoning queries?

RQ5: Knowledge Graph Augmentation

What does fusing an IDF-weighted knowledge-graph entity/relation signal into RRF change in IR metrics and structural concept coverage?

RQ6: Factorial Ablation Contributions

What are the isolated marginal contributions of Graph traversal, Deduplication, and MMR when evaluated through a full factorial design?

RQ7: ANN Vector Indexing Parity

Does an HNSW-based approximate nearest neighbor index (Qdrant) reproduce exact-search dense retrieval fidelity at scale?

RQ8: Calibrated Confidence (C3)

How can retrieval-native signals (rank, fusion score, lexical overlap) be leveraged for calibrated confidence estimation without auxiliary LLM latency?

03. Decoupled 3-Pipeline System Architecture

The architecture is engineered around three strictly decoupled pipelines: Ingestion, Retrieval, and Generation, communicating through strongly-typed schemas and persistent state caches.

Pipeline Stage	Core Responsibilities	Architectural Components & Tooling
1. Ingestion Pipeline	Multi-backend extraction, sentence-aware chunking preserving section headings, parameter-hash drift detection, heuristic knowledge-graph entity extraction, and dual storage persistence.	• `pypdfium2` (primary), `pdfplumber` (tables), `pypdf` (fallback) • Structured chunker (`max_words=200, overlap_sentences=1`) • SHA-256 parameter state cache (`.cache/`) • NetworkX knowledge graph with IDF-weighted degree activation • Partitioned Apache Parquet & Qdrant vector database
2. Retrieval Pipeline	Unified strategy dispatch across 18 configurations, candidate generation, score normalization, rank fusion, wide-pool cross-encoder reranking, and diversity postprocessing.	• Sparse: `rank-bm25` (BM25Okapi), `scikit-learn` TF-IDF • Distributional: Zero-dependency PPMI matrix factorization • Dense: `all-MiniLM-L6-v2` & `specter2_base` + proximity adapter • Cross-Encoder: `ms-marco-MiniLM-L-6-v2` (top-50 pool) • Graph: 1-hop traversal with Louvain community detection fallback • Fusion: Convex combination & Reciprocal Rank Fusion (`k=60`) • Postprocessing: Sliding-window Jaccard dedup & MMR (λ=0.7)
3. Generation Pipeline	Provenance-injected context formatting, grounding-constrained prompt construction, multi-provider frontier LLM routing, token bucket rate limiting, and Per-Claim Calibrated Confidence (C3).	• Context builder with `[Source N: doc.pdf \| Page P \| § Section]` • Provider adapters: OpenAI (`gpt-5.5`), Anthropic (`claude-sonnet-5`), Google Gemini (`gemini-3.8-flash`) • Dual-metered `TokenBucket` client-side rate limiting • Per-Claim Calibrated Confidence (C3) triaging (HIGH / MED / LOW)

04. The 18 Evaluated Retrieval Strategies

All 18 strategies execute through a unified dispatch interface (get_strategy_rankings), ensuring identical candidate evaluation environments:

#	Strategy Alias	Mechanism	Key Architectural Parameters
1	`bm25`	Okapi BM25 sparse keyword matching	k1 = 1.5, b = 0.75
2	`tfidf`	Sublinear TF-IDF vector space with cosine similarity	sublinear_tf = True
3	`linear_0.3`	Convex combination of normalized BM25 & TF-IDF	α = 0.3 (sparse-heavy)
4	`linear_0.5`	Convex combination of normalized BM25 & TF-IDF	α = 0.5 (equal blend)
5	`linear_0.7`	Convex combination of normalized BM25 & TF-IDF	α = 0.7 (dense-heavy)
6	`rrf`	Reciprocal Rank Fusion of BM25 + TF-IDF	k = 60
7	`rrf_dedup`	RRF followed by Jaccard sliding-window deduplication	threshold = 0.65
8	`rrf_dedup_mmr`	Deduplicated RRF re-ranked by Maximal Marginal Relevance	λ = 0.7, top_k = 5
9	`ppmi`	Zero-dependency PPMI distributional co-occurrence fused with BM25	window = 5, vocab = 1500
10	`cross_encoder`	`ms-marco-MiniLM-L-6-v2` re-ranks wide candidate pool	pool_size = 50
11	`sentence_transformer`	`all-MiniLM-L6-v2` dense bi-encoder with cosine similarity	384 dimensions
12	`adaptive`	Query-intent heuristic dynamic α weighting	Rule-based intent classifier
13	`specter2`	`allenai/specter2_base` with proximity adapter & CLS pooling	Asymmetric query adapter
14	`rrf_graph_dedup_mmr`	BM25 + TF-IDF + 1-hop NetworkX Graph fused via RRF → Dedup → MMR	IDF-weighted entity activation
15	`qdrant`	Approximate Nearest Neighbor vector search on Qdrant HNSW index	HNSW M=16, ef=100
A	`graph_only`	Standalone 1-hop NetworkX graph traversal with IDF activation	Isolates standalone KG signal
B	`rrf_graph`	RRF fusing BM25 + TF-IDF + Knowledge Graph (no dedup, no MMR)	Isolates raw graph fusion
C	`rrf_graph_dedup`	RRF + Knowledge Graph + Jaccard Dedup (no MMR)	Isolates Dedup without MMR

05. Comprehensive Quantitative Benchmark Results

Evaluated across 22 queries, 44 scientific documents, and 9,558 structured chunks. Ground truth is defined as a target chunk index per query complemented by curated target entity and relation lists:

#	Strategy Name	MRR	Recall@1	Recall@3	Recall@5	NDCG@5	EntCov	RelCov	Key Characteristic
1	Pure BM25 (Sparse)	0.551	0.364	0.727	0.773	0.589	0.902	0.727	Exceptional keyword precision on domain jargon
2	Pure TF-IDF (Sparse Vector)	0.337	0.136	0.409	0.591	0.373	0.826	0.636	Sparse vector space baseline with sublinear TF
3	Linear Hybrid (α=0.3)	0.503	0.273	0.682	0.773	0.556	0.902	0.727	Best linear blend; strongly weights sparse signal
4	Linear Hybrid (α=0.5)	0.503	0.273	0.773	0.773	0.558	0.886	0.773	Equal convex score combination
5	Linear Hybrid (α=0.7)	0.471	0.273	0.727	0.773	0.535	0.871	0.727	Dense-heavy blend; degraded by dense score noise
6	RRF (k=60)	0.514	0.318	0.727	0.773	0.566	0.856	0.682	Rank-space fusion immune to score-scale distortion
7	RRF + Deduplication	0.477	0.318	0.636	0.682	0.521	0.856	0.682	Eliminates redundant sliding-window chunk overlap
8	RRF + Dedup + MMR	0.486	0.318	0.682	0.727	0.547	0.902	0.727	Top ranking diversity via MMR (λ=0.7)
9	PPMI Semantic + BM25 RRF	0.373	0.182	0.455	0.591	0.401	0.856	0.727	Distributional co-occurrence semantics from scratch
10	Cross-Encoder Re-rank	0.481	0.318	0.591	0.773	0.549	0.902	0.773	Re-ranks 50 un-deduplicated candidates via MS MARCO
11	Sentence-Transformer (MiniLM)	0.318	0.182	0.364	0.500	0.335	0.818	0.682	Pure dense bi-encoder; diffuses coined technical terms
12	Adaptive Hybrid	0.491	0.273	0.682	0.727	0.533	0.886	0.773	Dynamic query-intent alpha weighting heuristic
13	SPECTER2 (Scientific Bi-Encoder)	0.278	0.136	0.364	0.500	0.327	0.871	0.682	Domain-adapted SciBERT with dual proximity adapters
14	RRF + Graph + Dedup + MMR	0.477	0.364	0.591	0.682	0.528	0.902	0.727	Fuses IDF-weighted NetworkX KG; high structural coverage
15	Qdrant Vector (ANN)	0.318	0.182	0.364	0.500	0.335	0.818	0.682	Sub-millisecond ANN vector search via Qdrant HNSW
A	Ablation: Graph only	0.106	0.045	0.091	0.273	0.146	0.417	0.273	Baseline: graph signal alone without lexical/dense fusion
B	Ablation: RRF + Graph	0.514	0.364	0.636	0.773	0.565	0.871	0.773	Isolates raw graph contribution to RRF fusion
C	Ablation: RRF + Graph + Dedup	0.477	0.364	0.545	0.682	0.519	0.871	0.773	Isolates Dedup contribution on top of graph fusion

06. Multi-Hop Reasoning Benchmark & Cross-Encoder Discovery

A pivotal empirical finding emerged when isolating the 6 complex multi-hop reasoning queries (*"How do POMDP belief states update after receiving new observations"*, *"What risk-tiering mechanisms does the AgentRunner framework apply"*, *"What market forces shape the organization and size of AI agent firms"*).

On multi-hop reasoning, the ranking dynamics completely inverted:

#	Strategy Name	MRR	Recall@1	Recall@3	Recall@5	NDCG@5	Key Reasoning Advantage
10	Cross-Encoder Re-rank ★	0.408	0.167	0.500	0.833	0.513	Best for multi-hop reasoning via joint token-level cross-attention
1	Pure BM25 (Sparse)	0.449	0.333	0.500	0.667	0.488	Strong lexical precision on composite query keywords
8	RRF + Dedup + MMR	0.367	0.167	0.500	0.667	0.441	MMR diversifies multi-step evidence chunks
6	RRF (k=60)	0.345	0.167	0.500	0.500	0.355	Pure rank fusion loses precision on relational queries
14	RRF + Graph + Dedup + MMR	0.306	0.167	0.500	0.500	0.355	Structural traversal surfaces broad relational context

The Dual-Strategy Architectural Division of Labor

Cross-encoders suffer from domain divergence on single-hop technical definitions because they were trained on general MS MARCO web queries. However, on complex multi-hop queries, their joint query-document cross-attention allows them to evaluate relational dependencies between disparate clauses that bi-encoders and keyword matchers fail to bridge. Recommendation: Use fast sparse/RRF retrieval for direct lookup, and route multi-hop or inferential queries to cross-encoder rerankers.

07. Factorial Graph-RAG Ablation Analysis

To dissect the interaction between knowledge graph traversal, sliding-window deduplication, and MMR, nine experimental cells were evaluated under identical benchmark queries:

Standalone KG Signal is Weak

graph_only (MRR = 0.106) demonstrates that entity graph traversal alone cannot replace textual retrieval; it functions as an associative amplifier, not a primary search engine.

Graph Elevates Top-1 Recall

Adding the Knowledge Graph to RRF increases Recall@1 from 0.318 to 0.364 while maintaining 0.514 MRR, proving structural entity links elevate relevant documents to position #1.

Deduplication Eliminates Churn

Jaccard deduplication prunes adjacent overlapping chunks, slightly trading raw single-chunk MRR (0.514 → 0.477) to eliminate redundant tokens from entering the generation context window.

MMR Recovers Quality

Maximal Marginal Relevance lifts NDCG@5 from 0.521 to 0.547 for non-graph and 0.519 to 0.528 for graph-augmented pipelines, maximizing informational diversity.

08. Lossless Qdrant HNSW ANN Vector Search Parity

Strategy 15 evaluated whether deploying an HNSW approximate nearest neighbor (ANN) vector database introduces accuracy degradation compared to exact matrix cosine similarity (Strategy 11) on 9,558 chunks:

Evaluation Dimension	Strategy 11: Exact Bi-Encoder (MiniLM)	Strategy 15: Qdrant HNSW ANN	Parity Assessment
Mean Reciprocal Rank (MRR)	0.318	0.318	100% Identical
Recall@1 / Recall@5	0.182 / 0.500	0.182 / 0.500	100% Identical
NDCG@5	0.335	0.335	100% Identical
Execution Latency & Scaling	In-memory O(N) linear brute-force scan	Sub-millisecond HNSW graph traversal (M=16, ef=100)	Production Scalable

09. Calibrated Confidence (C3) & Frontier Generation

To eliminate hallucination risks without incurring auxiliary LLM inference costs or latency, the architecture formalizes Per-Claim Calibrated Confidence (C3) derived directly from retrieval-native signals:

HIGH Confidence

rank ≤ 2 ∧ fusion_score ≥ 0.7 ∧ lexical_overlap ≥ 0.3
Direct citation grounding; zero hesitation in synthetic generation.

MEDIUM Confidence

rank ≤ 4 ∨ (fusion_score ≥ 0.5 ∧ lexical_overlap ≥ 0.2)
Qualified claim; passage provided with explicit provenance bracket.

LOW Confidence

Fails both criteria; passage flagged as speculative or discarded before generation to prevent ungrounded hallucinations.

10. Open Source Code & Academic Citation

The complete experimental suite, evaluation benchmarks, ingestion engines, and retrieval strategies are open source on GitHub:

Puri, A. (2025). Hybrid Retrieval Strategies for Retrieval-Augmented Generation over Scientific PDF Corpora: An Empirical Comparison. Systems & Empirical Information Retrieval Research. GitHub: https://go.dramitpurifoundation.org/Hybrid-Search-Repo

Full Research Paper

Read the complete empirical research paper covering all 18 retrieval strategies, mathematical formulations, and evaluation telemetry.

Read Research Paper

GitHub Open Source

Access the Python package, CLI tools, evaluation benchmark scripts, and reproduction guides.

Explore on GitHub

Companion Doctoral Study

Explore the complementary doctoral research (DBA Data Science) on Cloud Transformation Challenges & Low-Code platforms.

Read LCNC Research

Code Smells in Microservices

Explore the natural-experiment MS in Data Science study analyzing 754 monolith versions vs. microservices.

Read Code Smells Study

Hybrid Search & RAG Architecture