Building Production-Ready RAG Pipelines

Every RAG demo looks the same: embed some PDFs, throw them in a vector store, wire up a similarity search, and let an LLM answer questions over the results. It works beautifully for the first fifty queries. Then real users show up with real questions, and the whole thing starts hallucinating with confidence.

I've now shipped three RAG systems into production at Soletechnix, and the gap between "works in a notebook" and "works when someone's business depends on it" is almost entirely about the boring parts nobody puts in the demo.

The retrieval step is the product

Most teams spend 90% of their effort tuning the prompt and 10% on retrieval. That ratio should be inverted. If the wrong chunks get retrieved, no amount of prompt engineering saves you — the model is doing exactly what you asked: answering faithfully from bad context.

Three things moved the needle more than any prompt tweak ever did:

• Chunking by structure, not by character count. Splitting a technical doc every 500 characters cuts sentences and tables in half. Chunk along headings, list boundaries, and code blocks instead.
• Hybrid search, not pure vector similarity. Dense embeddings are great at "what is this about" and terrible at exact identifiers — SKU numbers, error codes, function names. Combine a keyword/BM25 pass with vector search and re-rank the merged set.
• Re-ranking before generation. A cheap cross-encoder re-ranker on the top 20 candidates, cutting down to the top 5, consistently outperformed increasing top_k on the vector store alone.

def retrieve(query: str, k: int = 5):
    dense_hits = vector_store.search(embed(query), top_k=20)
    sparse_hits = bm25_index.search(query, top_k=20)
    candidates = merge_and_dedupe(dense_hits, sparse_hits)
    return reranker.rerank(query, candidates)[:k]

Grounding, not just retrieval

Retrieval solves "did we find relevant text." It doesn't solve "did the model actually use it correctly." I added a lightweight grounding check as a second pass: after generation, verify every factual claim in the answer against the retrieved chunks with a smaller, cheaper model. If a claim can't be traced back to a source, either drop it or flag it for the user.

The single biggest trust-killer in a RAG product isn't retrieval failure — it's a confident answer with no traceable source. Users forgive "I don't know." They don't forgive a wrong answer stated as fact.

What I'd tell myself a year ago

1. Log every query, every retrieved chunk, and every generated answer from day one. You cannot debug retrieval quality from vibes.
2. Build an eval set of real questions before you ship, not after users complain.
3. Treat your chunking strategy as a first-class engineering decision, not a preprocessing afterthought.

RAG is not "add an LLM on top of search." It's a retrieval system with a language model as one component — and it should be engineered with the same rigor you'd give any other search infrastructure.