RAG Reranking: The One Layer Most Teams Skip That Kills Precision

Most RAG pipelines retrieve 20 chunks and hand all 20 to the LLM. That's not retrieval-augmented generation — that's context-window stuffing with extra steps. The precision problem doesn't live in your vector index. It lives in the gap between retrieved and relevant.

Reranking is the layer that closes that gap. It's also the layer that gets cut when a team is moving fast and the demo looks good enough. Here's why that decision degrades badly in production, and what to actually build instead.

Why Your Top-K Retrieval Score Is a Weak Signal

Vector similarity (cosine similarity on normalized embeddings, or dot product if your index handles normalization) measures one thing: geometric proximity in embedding space. That's a proxy for semantic relatedness, not answer quality. A chunk that contains the exact keyword your query uses but provides zero useful context will often outscore a chunk that actually answers the question but uses different vocabulary.

This is the core failure mode: embedding models are trained to cluster semantically related text, not to rank by answerability. The scores your ANN index returns — whether from FAISS, Pinecone, Weaviate, or Qdrant — are distance metrics, not relevance judgments. Teams treat them as relevance judgments and then wonder why their RAG system hallucinates on questions where the answer is clearly in the corpus.

The gap gets worse as your corpus grows. At 10,000 chunks, a vector search returning the top 5 is probably fine. At 500,000 chunks across a heterogeneous document set — mixed formats, mixed domains, varying chunk sizes — the signal-to-noise ratio in your top-20 retrieval results degrades significantly. To make that concrete: imagine a legal document corpus where a query about "termination clauses" returns 20 chunks, but chunks 3 and 7 are the ones that actually answer the question — buried under superficially similar but useless boilerplate. You need a second-pass scorer that understands the query-document relationship, not just document proximity.

What Reranking Actually Does (and What It Doesn't)

A reranker is a cross-encoder: it takes a (query, document) pair as a single input and outputs a relevance score. Unlike bi-encoders (your embedding model), which encode query and document independently and compare vectors, a cross-encoder processes them jointly — attention flows across both — so it can model fine-grained interactions like negation, specificity, and conditional relevance.

The tradeoff is speed. A bi-encoder retrieves in milliseconds because you pre-compute document embeddings and do a single vector lookup. A cross-encoder runs inference on every (query, chunk) pair at query time. At top-20 retrieval → rerank → top-5 output, you're running 20 forward passes. This is why you don't replace your vector search with a reranker — you use vector search to get a candidate set (fast, cheap, imperfect) and the reranker to score that candidate set (slower, expensive, precise).

Practical latency reality:

• Bi-encoder retrieval: 10–50ms for most hosted vector DBs at reasonable scale
• Cross-encoder reranking on 20 chunks: 150–400ms for a model like cross-encoder/ms-marco-MiniLM-L-6-v2 running on a small GPU instance; 50–150ms for Cohere Rerank API or similar hosted endpoints
• End-to-end with LLM generation: reranking adds 10–25% to total latency in most pipelines, which is worth it for any application where answer quality matters

For real-time voice agents where every millisecond matters, that latency budget is a different conversation. For async or near-real-time text applications, the tradeoff is almost always worth making.

Three Reranking Patterns Worth Knowing

1. Hosted Rerank API (Cohere, Jina, Voyage) Fastest path to production. Cohere's Rerank endpoint, Jina's reranker, and Voyage AI's reranker are all drop-in API calls. You send your query and retrieved chunks, get back scores and reordered results. No GPU infra, no model management. Cost is low — check current provider pricing pages before budgeting, as rates shift — and latency is competitive. Start here unless you have a hard data-residency requirement.

2. Self-hosted Cross-Encoder For teams with data sovereignty requirements (common in MENA financial and government deployments), running a model like BAAI/bge-reranker-v2-m3 or cross-encoder/ms-marco-MiniLM-L-12-v2 on your own infrastructure is the path. A single A10G handles the reranking load for most mid-scale applications comfortably. The open-source model quality is close to — not identical to — the hosted options for general English. For Arabic or multilingual corpora, BAAI/bge-reranker-v2-m3 is currently the best open-weight option I'm aware of.

3. LLM-as-Reranker (RankGPT pattern) Pass your top-K chunks to an LLM and ask it to reorder them by relevance before generating. This sounds expensive — and it is — but it can work well in offline indexing pipelines where you're pre-computing relevance for a known query distribution, or in high-stakes low-volume use cases where precision matters more than cost. Don't use this in your hot path.

The Retrieval Pipeline That Actually Works at Scale

Here's the architecture I wire up for production RAG systems:

python
# Simplified pipeline sketch — not production error handling
def retrieve_and_rerank(query: str, top_k_retrieve: int = 20, top_k_final: int = 5):
    # Stage 1: Fast ANN retrieval from vector store
    candidates = vector_store.similarity_search(query, k=top_k_retrieve)
    # Returns list of (chunk_text, metadata, vector_score)

    # Stage 2: Cross-encoder reranking
    rerank_inputs = [(query, chunk.page_content) for chunk in candidates]
    scores = reranker.predict(rerank_inputs)  # cross-encoder inference

    # Stage 3: Sort by rerank score, take top_k_final
    ranked = sorted(zip(scores, candidates), key=lambda x: x[0], reverse=True)
    return [chunk for _, chunk in ranked[:top_k_final]]

The key decisions in this pipeline:

• top_k_retrieve should be 3–5x your final top_k. You need enough candidates for the reranker to have real signal. Too few and you've already thrown away the right answer before reranking starts.
• Don't pass rerank scores to the LLM. Just use the ordering. The LLM doesn't need the score — it needs the right chunks in the right positions (leading chunks get more attention weight in practice).
• Add a minimum score threshold. If your top reranked chunk scores below, say, 0.2 on a 0–1 scale, you may be better off returning "I don't have enough information" than hallucinating an answer from loosely related content.

Decision Table: When to Add Reranking

Situation	Add Reranking?	Why
Corpus < 10k chunks, single domain	Maybe not	Retrieval signal is usually sufficient
Corpus > 50k chunks, mixed domains	Yes	False positives multiply with scale
High-stakes answers (legal, medical, financial)	Always	Cost of wrong answer exceeds rerank latency
Real-time voice (< 300ms budget)	No — optimize elsewhere	Latency too tight; fix chunking strategy instead
Multilingual corpus	Yes, with multilingual reranker	Embedding similarity degrades cross-language
User-facing enterprise search	Yes	Users notice precision gaps immediately

For teams building hybrid retrieval (vector + BM25 keyword), reranking is even more important — you're combining scores from two different systems that aren't on the same scale. The reranker acts as a unified scorer across both retrieval channels, normalizing signals that are otherwise incommensurable.

The Two Mistakes Teams Make After Adding Reranking

Mistake 1: Shrinking top_k_retrieve to "save money" The retrieval stage is cheap. Embedding lookup at scale costs fractions of a cent per query on any hosted vector DB. Teams see the reranker's per-call cost and try to reduce the number of chunks it processes by retrieving fewer candidates. This defeats the purpose — you're starving the reranker of the candidates it needs to surface the best result. Keep top_k_retrieve generous; control cost at the LLM generation stage instead.

Mistake 2: Never evaluating rerank quality offline You can't tune what you don't measure. The minimum viable evaluation setup: take 50–100 representative queries, manually label which chunks in your corpus are relevant, and compute NDCG@5 (normalized discounted cumulative gain) before and after reranking. If your reranker isn't improving NDCG@5 by a meaningful margin, either your chunk boundaries are wrong, your embedding model is too weak for the domain, or your reranker model doesn't fit the language/domain distribution. Each of those has a different fix.

Here's a more complete runnable example using sklearn:

python
from sklearn.metrics import ndcg_score
import numpy as np

# relevance_labels: dict mapping chunk_id -> relevance grade (0=not relevant, 1=partial, 2=highly relevant)
# retrieved_ids: ordered list of chunk IDs returned by your pipeline

def evaluate_ranking(retrieved_ids: list, relevance_labels: dict, k: int = 5) -> float:
    """
    Compute NDCG@k for a single query.
    retrieved_ids: the ranked list of chunk IDs your pipeline returned
    relevance_labels: ground-truth relevance grades for those IDs
    k: cutoff (typically 5)
    """
    # Truncate to k and look up true grades
    top_k_ids = retrieved_ids[:k]
    true_relevance = [relevance_labels.get(cid, 0) for cid in top_k_ids]

    # Ideal order: sort grades descending
    ideal = sorted(true_relevance, reverse=True)

    # ndcg_score expects shape (n_queries, n_docs)
    score = ndcg_score([ideal], [true_relevance])
    return score


# Example: 5 retrieved chunks, manually graded
example_ids      = ["chunk_03", "chunk_17", "chunk_08", "chunk_22", "chunk_05"]
example_labels   = {"chunk_03": 0, "chunk_17": 2, "chunk_08": 1, "chunk_22": 0, "chunk_05": 2}

ndcg = evaluate_ranking(example_ids, example_labels, k=5)
print(f"NDCG@5: {ndcg:.4f}")
# Output will be < 1.0 because the highest-relevance chunks aren't ranked first
# A perfect ranking of [chunk_17, chunk_05, chunk_08, chunk_03, chunk_22] would yield 1.0


# To compare pre- and post-rerank across a query set:
def batch_ndcg(query_results: list[dict], k: int = 5) -> float:
    """
    query_results: list of {"retrieved_ids": [...], "labels": {...}}
    Returns mean NDCG@k across all queries.
    """
    scores = [
        evaluate_ranking(r["retrieved_ids"], r["labels"], k)
        for r in query_results
    ]
    return float(np.mean(scores))

Run batch_ndcg on both your pre-rerank and post-rerank orderings across your 50–100 query eval set. The delta tells you whether the reranker is earning its latency cost. In practice, a well-configured reranker on a heterogeneous corpus should move NDCG@5 noticeably — if it doesn't, the issue is upstream (chunking, embedding model fit, or domain mismatch), not the reranker itself.

What to Actually Do

1. Audit your current top_k setting today. If you're retrieving fewer than 10–15 candidates before generation, your retrieval ceiling is artificially low regardless of reranking.
2. Start with a hosted reranker (Cohere Rerank or Voyage). Get the quality signal first. Swap to self-hosted later if data residency requires it. Check current pricing directly with the provider before budgeting.
3. Build a 50-query evaluation set before you ship. Label relevance manually or with a capable LLM judge. Compute NDCG@5 using the code above. This takes a day, not a sprint.
4. Set a minimum score threshold for citation. If the top chunk doesn't clear your threshold, surface that uncertainty to the user rather than hallucinating confidence.
5. For multilingual or Arabic-heavy corpora, test BAAI/bge-reranker-v2-m3 explicitly. General English rerankers degrade measurably on Arabic and mixed-script content.

Retrieval gets you in the room. Reranking decides who speaks.