Technology Trends
7 Techniques to Sharpen RAG Retrieval Accuracy in Your LlamaIndex Python App
Your LLM is not the problem. Nine times out of ten, when a RAG pipeline spits out vague, hallucinated, or off-topic answers, the model gets the blame — but the real culprit is the three seconds before the model ever sees anything: retrieval. Fix what goes into the context window, and the answers fix themselves.
This post covers 7 concrete techniques — chunking strategies, indexing approaches, and reranking methods — that you can wire into a Python RAG application using LlamaIndex today. Each one targets a specific failure mode. These techniques apply whether you’re pushing inference through vLLM on a GPU server or doing Python local LLM development with Ollama on a laptop — the retrieval layer is where accuracy is won or lost either way. By the end, you’ll know which technique to reach for and when.
Why Retrieval Quality Is the Bottleneck
Think of the LLM as a very smart person who can only answer based on the notes you hand them. Hand them the wrong notes, and no amount of intelligence helps.
That’s the retrieval-generation contract in practice: the model is bounded by whatever context it receives. A well-tuned retrieval pipeline consistently outperforms a larger model paired with a poorly tuned one. I’ve watched teams swap GPT-3.5 for GPT-4 and see zero improvement — because they were still handing the model the wrong chunks.
Here’s where things typically go wrong:
- Semantic drift — the embedding of the query matches documents that are topically adjacent but not actually relevant
- Chunk boundary cuts — a key sentence gets split across two chunks; neither chunk alone is useful
- Over-retrieval — too many chunks dilute the signal; the answer is buried in noise
- Under-retrieval — too few chunks miss the evidence entirely
LlamaIndex models its pipeline as: VectorStoreIndex → QueryEngine → NodePostprocessor chain → LLM. Each layer is a place where you can intervene. The techniques below map to specific layers.
User Query → [Retriever] → [Postprocessors / Rerankers] → [LLM] → AnswerTechniques 1 & 2: Smarter Chunking Strategies
Before a single query runs, the shape of your chunks determines what’s even possible to retrieve. Most tutorials stop at “split on 512 tokens.” That’s rarely the right answer.
Technique 1: Sentence-Window Chunking
The idea is elegant: index small, sentence-level chunks for precise embedding, but at query time, replace each retrieved chunk with a surrounding window of sentences before passing context to the LLM.
Tight semantic matching during retrieval. Enough surrounding context for the model to reason. Both at once.
from llama_index.core.node_parser import SentenceWindowNodeParser
from llama_index.core.postprocessor import MetadataReplacementPostProcessor
node_parser = SentenceWindowNodeParser.from_defaults(
window_size=3, # sentences on each side
window_metadata_key="window",
original_text_metadata_key="original_text",
)
# Attach during query engine construction
postprocessor = MetadataReplacementPostProcessor(
target_metadata_key="window"
)
query_engine = index.as_query_engine(
node_postprocessors=[postprocessor]
)This works particularly well on legal documents, academic papers, or technical manuals — anywhere individual sentences carry independent meaning but context is everything.
Technique 2: Hierarchical / Auto-Merging Chunking
Here’s the thing most chunking guides skip: sometimes the problem isn’t what you retrieve, it’s that you retrieve fragments of the same logical block separately. The answer spans a paragraph, but you get three disconnected sentence-chunks back.
Auto-merging chunking solves this. You store leaf nodes (small chunks), but if enough sibling chunks from the same parent are retrieved, LlamaIndex merges them back into the parent before handing off to the LLM.
from llama_index.core.node_parser import HierarchicalNodeParser, get_leaf_nodes
from llama_index.core.retrievers import AutoMergingRetriever
node_parser = HierarchicalNodeParser.from_defaults(
chunk_sizes=[2048, 512, 128] # parent → child → leaf
)
nodes = node_parser.get_nodes_from_documents(documents)
leaf_nodes = get_leaf_nodes(nodes)
# Build index on leaf nodes only
storage_context.docstore.add_documents(nodes)
base_retriever = index.as_retriever(similarity_top_k=12)
retriever = AutoMergingRetriever(
base_retriever, storage_context, verbose=True
)Use this whenever your documents have a natural section/subsection hierarchy — user manuals, runbooks, specification docs.
Techniques 3 & 4: Indexing Strategies That Improve Recall
A flat vector index is a reasonable starting point. It’s rarely the finishing point.
Technique 3: Hybrid Search (Dense + Sparse)
Hybrid search is a retrieval approach that combines dense vector similarity with BM25 sparse keyword matching — giving you semantic understanding for conceptual queries and exact-match precision for terms that embeddings tend to blur.
And embeddings do blur things. Ask a dense-only retriever for “error code ER-4291” and it might return something conceptually adjacent but factually wrong. BM25 handles exact-match keyword queries the way search engines have for decades. Combining the two is one of the highest-ROI changes you can make to a retrieval pipeline.
from llama_index.core.retrievers import QueryFusionRetriever, BM25Retriever
vector_retriever = index.as_retriever(similarity_top_k=10)
bm25_retriever = BM25Retriever.from_defaults(
nodes=nodes, similarity_top_k=10
)
hybrid_retriever = QueryFusionRetriever(
retrievers=[vector_retriever, bm25_retriever],
similarity_top_k=5,
mode="reciprocal_rerank",
use_async=True,
)| Query Type | Dense-Only Accuracy | Hybrid Accuracy |
|---|---|---|
| Semantic / conceptual | High | High |
| Exact product codes / IDs | Low | High |
| Named entities / proper nouns | Medium | High |
When you’re optimizing for latency — say, running Python LLM inference optimization with vLLM in production — hybrid search with use_async=True keeps the two retrieval paths parallel, minimizing added overhead.
Technique 4: Metadata Filtering
I once saw a team spend two weeks tuning embeddings on a multi-tenant knowledge base, confused about why irrelevant documents kept surfacing. The problem was simpler: they had no metadata filters, so every tenant’s queries searched every tenant’s documents.
Attaching metadata at ingest and filtering at query time narrows the search space before semantic similarity even runs.
from llama_index.core.vector_stores import MetadataFilter, MetadataFilters
filters = MetadataFilters(filters=[
MetadataFilter(key="tenant_id", value="acme_corp"),
MetadataFilter(key="doc_version", value="v2.1"),
])
retriever = index.as_retriever(
similarity_top_k=10,
filters=filters,
)This is low-hanging fruit. If your corpus has any categorical structure — by source, date range, document type, or user — filter it. The precision gain is immediate.
Techniques 5 & 6: Query-Side Enhancements
The query itself is a retrieval input you can engineer. Most pipelines treat it as immutable. They shouldn’t.
Technique 5: HyDE — Hypothetical Document Embeddings
Short user queries embed poorly. “What’s the refund window?” is a question; your documents contain answers. The embedding spaces don’t naturally align.
HyDE (Hypothetical Document Embeddings) is a query transformation technique that generates a hypothetical answer to the question first, then embeds that answer for retrieval instead of the raw query — bridging the gap between how users ask questions and how documents are written.
from llama_index.core.indices.query.query_transform import HyDEQueryTransform
from llama_index.core.query_engine import TransformQueryEngine
hyde_transform = HyDEQueryTransform(include_original=True)
hyde_query_engine = TransformQueryEngine(query_engine, hyde_transform)
response = hyde_query_engine.query("What's the refund window?")The include_original=True keeps the original query in the mix — a useful hedge when HyDE’s hypothetical drifts from intent. Use this on short queries and domain-specific corpora where user vocabulary doesn’t match document vocabulary.
Technique 6: Multi-Query Retrieval and Result Fusion
Different phrasings of the same question retrieve different documents. That’s not a bug — it reflects genuine ambiguity in embedding space. So generate multiple phrasings, retrieve for each, deduplicate, and fuse.
from llama_index.core.retrievers import QueryFusionRetriever
retriever = QueryFusionRetriever(
retrievers=[vector_retriever],
llm=llm,
num_queries=4,
mode="reciprocal_rerank",
use_async=True,
)This adds LLM calls per query. If cost is a concern, run the rewriting step through a local model. Python local LLM development with Ollama makes this straightforward — you’re not paying per token for what’s essentially a paraphrasing task, and a 7B model handles it fine.
Technique 7: Reranking Before Generation
Retrieval casts a wide net. Reranking sharpens it.
Why Top-K Order Matters
Reranking is a postprocessing step that rescores retrieved nodes against the query using a cross-encoder model — one that reads query and document together, producing more accurate relevance judgments than cosine similarity alone.
Research on the “lost-in-the-middle” problem (Liu et al., 2023) showed that LLMs perform significantly better when relevant content appears at the beginning or end of their context, not buried in the middle. So the order of retrieved chunks matters, not just which chunks you retrieve.
Implementing a Reranker in LlamaIndex
from llama_index.core.postprocessor import SentenceTransformerRerank
reranker = SentenceTransformerRerank(
model="cross-encoder/ms-marco-MiniLM-L-6-v2",
top_n=3,
)
query_engine = index.as_query_engine(
similarity_top_k=15, # retrieve wide, rerank narrow
node_postprocessors=[reranker],
)Retrieve 15, rerank to 3. That pattern works well. The cross-encoder sees all 15 candidates relative to the query and picks the genuinely best ones.
For higher-throughput scenarios, CohereRerank is a solid API-based option. If you’re already serving models with vLLM, you can host a reranking model there too — vLLM-hosted cross-encoders give you throughput at scale without adding a separate service.
Stacking multiple rerankers is worth it only when your initial retrieval pool is large (50+ candidates). Below that, one good reranker is enough.
Building a Production Python RAG Application Using LlamaIndex
Not every project needs all seven from day one. But when you’re ready for a production-grade pipeline, here’s the layering order I’d recommend — and which problem each layer solves.
| Technique | Pipeline Layer | Problem It Solves | When to Add |
|---|---|---|---|
| Sentence-window chunking | Ingest | Fragmented context reaching the LLM | Dense, sentence-heavy documents |
| Auto-merging chunking | Ingest | Disconnected sibling chunks retrieved separately | Hierarchically structured documents |
| Hybrid search | Index | Exact-match terms missed by vector search | Any corpus with names, codes, or IDs |
| Metadata filtering | Retrieval | Irrelevant docs from the wrong scope | Multi-tenant or versioned corpora |
| HyDE | Query | Short or ambiguous user queries | Domain-specific vocabulary gaps |
| Multi-query fusion | Query | Single phrasing misses relevant documents | Semantically rich corpora |
| Cross-encoder reranking | Postprocessing | Wrong ordering of retrieved chunks | Large retrieval pools (10+ candidates) |
HyDE and sentence-window chunking are situational — add them when you’ve identified the specific failure mode they address, not by default.
Measure before and after with RAGAs or LlamaIndex’s built-in eval modules. Track hit rate (is the right document retrieved at all?) and NDCG (is it ranked highly?). Those two metrics will tell you exactly where your pipeline is leaking.
Frequently Asked Questions
What’s the easiest first technique to implement?
Metadata filtering. If your documents have any categorical structure — by source, date, or type — adding filters at query time immediately reduces noise without touching your indexing setup. It’s a one-liner addition to an existing query engine and requires no re-indexing.
Does hybrid search work with all vector stores?
Not all of them support it natively. Weaviate, Qdrant, and Elasticsearch have built-in hybrid search. For vector stores that don’t, LlamaIndex’s QueryFusionRetriever handles it at the application layer by running a separate BM25 retriever in parallel. The performance is slightly lower than native hybrid, but the approach works with any backend.
When should I use HyDE vs. multi-query retrieval?
They solve different problems. HyDE helps when the query itself is poorly formed for retrieval — short, ambiguous, or using vocabulary that doesn’t match your documents. Multi-query retrieval helps when the query is fine but a single phrasing doesn’t surface all relevant documents. In practice, multi-query is more broadly applicable. HyDE carries a higher risk of drifting from the user’s actual intent, so pair it with include_original=True as a safety net.
Does reranking always improve results?
Not always. If your initial retrieval pool is small and already high-quality, reranking adds latency without meaningful gain. The benefit is clearest when you’re retrieving a large candidate set (10+) and need to select the top 3-5 for the LLM’s context. Think of it as a precision filter layered on top of a recall-optimized retriever — it only helps when there’s something to filter.
Should I use vLLM or Ollama for query rewriting in multi-query retrieval?
It depends on where you are in the build cycle. If you’re already running vLLM for your main generation model, route the query-rewriting calls there too — the throughput handles it without breaking a sweat. If you’re earlier in development and don’t have a GPU server available, Ollama running a local 7B model does the job for paraphrasing at zero API cost. The quality difference for this specific task is negligible either way.
Wrapping Up
Every technique here targets something specific: chunk shape, index structure, query quality, or candidate selection. The teams that get stuck are usually the ones making general improvements — bigger models, more data, longer context — without diagnosing where the pipeline is actually failing.
Pick one failure mode. Apply the matching technique. Measure the delta. That’s the loop that moves the needle.
Start with your vector store’s retrieval logs, look at what’s coming back for your worst-performing queries, and you’ll know exactly where to begin. If you want a zero-cost environment to iterate, set up Ollama locally and test against a local model before wiring in anything API-based — the feedback loop is faster, and running 20 experiments costs nothing.
