RAG Is Not Retrieval: 5 Pipeline Mistakes Killing Accuracy

Most RAG systems that fail in production don't fail because the model is bad. They fail because someone treated retrieval as a solved problem and moved on. The retrieval pipeline is where your accuracy lives or dies — and most teams get at least three of these five things wrong.

Chunking Is an Engineering Decision, Not a Default Setting

Every vector store tutorial starts with chunk_size=512. That number is not neutral. It comes from a tradeoff between embedding model context limits, retrieval granularity, and the density of your documents — and 512 tokens is wrong for most real-world corpora.

Here's the decision rule:

• Short, fact-dense documents (FAQs, product specs, regulatory clauses): chunk small — 128–256 tokens with 20–30% overlap. You want one retrievable unit per atomic fact.
• Long-form narrative content (reports, contracts, manuals): chunk larger — 512–1024 tokens — but always at natural boundaries (paragraph or section breaks), never mid-sentence.
• Structured tabular data: don't embed the table; embed the row-level description you'd write if you were answering a question about that row.

Fixed-size character splitting — the default in most frameworks — ignores sentence and paragraph boundaries entirely. A chunk that starts mid-clause will embed as noise. Use a semantic or recursive text splitter, and run a quick sanity check: retrieve the top-5 chunks for 20 representative queries and read them. If any chunk is syntactically broken, your splitter is wrong.

Parent-child chunking is underused and worth understanding: index small child chunks for precision retrieval, but return the larger parent chunk to the LLM for context. You get recall-precision balance without a reranker. LlamaIndex calls this ParentDocumentRetriever; LangChain has the same pattern.

Cosine Similarity Alone Will Betray You in Production

Vector similarity is a proxy for relevance, not relevance itself. When you're using normalized embeddings, the dot product and cosine similarity are equivalent — but that's the least of your problems.

The real issue: embedding models collapse meaning in ways that destroy retrieval for domain-specific content. "The patient was discharged" and "the patient was admitted" have high cosine similarity in most general-purpose embedding spaces. For a healthcare RAG system, they mean the opposite thing.

Three failure modes that surface repeatedly in production:

1. No keyword fallback. Pure dense retrieval misses exact entity matches. A user asking about "IFRS 17" or "Article 42 of Cabinet Decision No. 52" needs that phrase matched precisely. Hybrid retrieval — dense + BM25 — with reciprocal rank fusion (RRF) is the standard fix. Elasticsearch, OpenSearch, and Weaviate all support this natively. Qdrant and Pinecone have hybrid search in their current releases.

2. No reranker. Your top-k retrieval returns 20 chunks; your LLM sees 5. The order matters enormously. A cross-encoder reranker (Cohere Rerank, BGE-Reranker, or Jina Reranker) re-scores chunk-query pairs using full attention — far more accurate than vector similarity alone. The latency cost is real: expect 100–300ms added per query depending on k and model size. Worth it for almost every non-latency-critical application.

3. Stale embeddings. Your documents updated; your index didn't. This is operational, not architectural — but it kills trust faster than anything else. Build incremental indexing from day one. Hash document chunks, store the hash alongside the vector, diff on ingest. Don't do full re-indexing unless schema changes force it.

python
import hashlib

def chunk_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()[:16]

# On ingest: only embed if hash has changed
def should_reindex(chunk_text: str, stored_hash: str) -> bool:
    return chunk_hash(chunk_text) != stored_hash

The Context Window Is Not a Trash Can

Teams that fix retrieval immediately go wrong on context assembly. They retrieve 20 chunks, shove them all into the prompt, and wonder why the model hallucinates or ignores half the evidence.

Two things that degrade badly here:

Lost-in-the-middle effect. Research from Liu et al. (2023) showed that LLMs reliably attend to content at the start and end of long contexts, and systematically underweight content in the middle. If your most relevant chunk lands at position 8 of 15, the model may not use it. Solution: sort retrieved chunks by relevance score descending before assembly, and cap context aggressively. Five well-ranked chunks outperform fifteen mixed-quality ones.

Context contamination. Irrelevant chunks don't just get ignored — they add noise that actively degrades answer quality. Set a minimum relevance threshold and drop chunks below it, even if that means fewer chunks. For normalized embeddings where cosine similarity and dot product are equivalent, a threshold around 0.75 is a reasonable starting point; calibrate against your domain.

Context assembly template worth adopting:

code
System: You are a [domain] assistant. Answer using ONLY the provided context. 
If the context does not contain the answer, say so.

Context:
[Chunk 1 — highest relevance]
[Chunk 2]
[Chunk 3]

Query: {user_query}

That instruction — "say so" — matters. Without it, models confabulate when retrieval fails. Hallucination in RAG is almost always a retrieval miss dressed up as a confident answer.

Metadata Filtering Is the Feature Nobody Ships

Vector similarity retrieves semantically similar content from your entire corpus. For most production systems, that's wrong. A customer asking about their specific account type shouldn't get chunks from a different product tier. A user querying 2024 regulations shouldn't get 2019 results ranked above current ones.

Metadata filtering — pre-filtering the vector search space before similarity scoring — is the fix. Every mature vector store supports it. What teams don't do is instrument their documents correctly at ingest time.

The metadata you should capture at ingest, at minimum:

Field	Type	Why It Matters
source_id	string	Deduplicate, trace provenance
created_at	datetime	Recency filtering
doc_type	enum	Scope by document category
language	string	Critical for multilingual corpora
version	string	Policy/regulatory versioning
access_level	enum	Entitlement-aware retrieval

Entitlement-aware retrieval is particularly important if you're building RAG over internal enterprise knowledge. You do not want a junior employee's query returning executive compensation documents because they're semantically similar to their HR question. Filter at retrieval time, not at display time. Filtering at display time means you've already spent tokens on content you can't show.

This intersects directly with data readiness: if your documents don't have clean metadata when they enter the pipeline, you can't filter them reliably later. Retrofit is painful. Do it upfront.

Evaluation Is Not Optional — It's How You Know the Pipeline Is Working

Most teams ship RAG, demo it to a stakeholder, get a thumbs up, and call it done. Then edge cases surface in production and there's no baseline to measure against. You don't know if a change you made improved things or made them worse.

RAG evaluation has three distinct layers — most teams only measure one:

Retrieval quality: Did the pipeline surface the right chunks? Metrics: Recall@k (is the gold chunk in your top-k?), Mean Reciprocal Rank (MRR), NDCG. You need a labeled evaluation set — 50–100 query/gold-chunk pairs. This takes time to build but it's the only ground truth you have.

Generation faithfulness: Did the LLM stick to the retrieved context, or did it add things that aren't there? Tools like RAGAS (open-source) measure faithfulness and answer relevance automatically using an LLM-as-judge pattern. Imperfect, but fast.

End-to-end answer correctness: Does the final answer match the ground truth? This requires human-labeled QA pairs. Start with 50 and expand. Use it for regression testing — any pipeline change should run against this set before it ships.

The teams that treat RAG evaluation as a one-time task are the ones whose systems degrade silently in production.

This compounds in multi-agent systems: if your retrieval layer is feeding stale or low-quality context into a downstream agent, you have no clean signal on where the failure originated. Instrument each layer independently before you wire them together.

Automate it. Run the eval suite on a schedule against your production index. Alert when Recall@5 drops below your threshold. Treat it like uptime monitoring.

What to Actually Do

1. Audit your chunking strategy this week. Pull 20 representative queries, retrieve top-5 chunks, read them. If any are syntactically broken or clearly irrelevant, rewrite your splitter before you touch anything else.

2. Add hybrid retrieval if you're pure-dense. BM25 + dense with RRF is a one-sprint change in most stacks and it handles exact entity matching that embedding models routinely miss.

3. Stand up a reranker. Cohere Rerank has a free tier. BGE-Reranker runs locally. Pick one, wire it between retrieval and context assembly, and measure the difference on your eval set.

4. Define your metadata schema and enforce it at ingest. Especially access level and document version. Retrofitting this later is the kind of work that stalls teams for weeks — not because the engineering is hard, but because nobody owns the decision.

5. Build a 50-query eval set and run it before every pipeline change. If you don't have a baseline, you're flying blind. This is non-negotiable before you call a RAG system production-ready.

RAG done well is genuinely powerful. But it's not plug-and-play — it's a pipeline with five or six places where the wrong default destroys accuracy. Fix the pipeline before you blame the model.