If you are searching for RAG implementation best practices, here is the short version: retrieval quality decides whether the system works, not which model sits on top. Everything else in this guide, chunking, hybrid search, re-ranking, caching, faithfulness gates, exists to make the right context reach the model reliably, and to make failures loud instead of silent.

I build retrieval systems for client work and for my own products. Two systems on this site carry the load of these lessons: Estate360, an agentic answer loop for real-estate workflows with confidence and faithfulness gates, and EngineerOS, a LangGraph multi-agent system whose retrieval test gates CI. This guide is what I would tell any team about to ship RAG to real users.

Retrieval quality matters more than model choice

Teams routinely spend weeks swapping models and days on retrieval. That is backwards. A frontier model with the wrong three chunks in context produces a confident wrong answer; a mid-tier model with the right three chunks produces a usable one. Before you touch the model, measure recall: does the context you retrieve actually contain the answer, and does it rank it near the top? If retrieval is broken, no model swap will save you, and if retrieval is good, model choice becomes a cost and latency decision rather than a correctness one.

Chunking: structure-aware, not character-count

Naive fixed-size chunking splits sentences in half and separates answers from the headings that define them. Structure-aware chunking splits on the document's own skeleton: headings, sections, list boundaries, table rows. Keep the section path as metadata on every chunk; it is cheap to store and it pays for itself in citation quality and in filtering.

As a documented working point, I use roughly 380-word chunks with a 50-word overlap. That is a starting position, not a law. The overlap exists so that an answer straddling a boundary is still retrievable whole. What matters is that you pick a point, document it, and re-measure when you change it, because every chunking decision ripples into your index, your embeddings and your evals.

Hybrid search and re-ranking

Pure vector search under-recalls exact identifiers. Product codes, ticket IDs, error strings, proper nouns, version numbers: embeddings blur these, because semantic similarity was never designed to respect an exact string. Keyword search (BM25 or Postgres full-text) nails them and misses paraphrase. You need both.

The architecture that works in production is: run vector and keyword retrieval in parallel, fuse the ranked lists (reciprocal rank fusion is the simple, robust option), then re-rank the fused candidates with a cross-encoder or a small model before stuffing context. Re-ranking is where most of the precision gain lives, because fusion alone leaves ordering noisy. In Postgres, this maps to pgvector for the dense side and full-text search with GIN indexes for the sparse side, fused in SQL or in application code. Keep the two indexes consistent with the same source-of-truth table.

Embeddings: comparable only within one model

Embeddings from different models live in different spaces. A vector from model A is meaningless to model B, even at the same dimensionality. The practical consequences: pin the embedding model in your config, version it, and record it with the index. If you switch embedders, you must re-embed and rebuild the index; there is no migration shortcut. Budget reindex time into any plan that includes "we might upgrade the embedding model later", which is most plans. Mixed-index querying across two embedders returns garbage quietly, which is worse than an error.

Evaluate with a fixed set of real questions, in CI

RAG degrades silently. A prompt tweak, a chunking change, an embedding bump, a new parser: any of these can drop recall without a single user-visible error until the complaints arrive. The fix is boring and effective: fix a set of real questions with known-good answers (20 is a workable start), run it in CI, and fail the build below a recall floor.

EngineerOS does exactly this. Its 20-question retrieval test fails the build if recall drops below its floor. That gate is the reason retrieval changes ship with confidence instead of with crossed fingers. Build the question set from actual support tickets, actual sales questions, actual internal docs queries. Synthetic questions written by the engineer who built the prompt will flatter the system every time. Include hard cases: ambiguous phrasing, identifiers, multi-hop questions, and questions that have no answer in the corpus.

Measure faithfulness, not just retrieval

Recall tells you the answer was in the context. Faithfulness tells you the model actually used it instead of freelancing. Both are necessary; only one is commonly measured. Estate360 gates at 0.8 faithfulness and 0.65 confidence, and below those gates it degrades to "I don't know" instead of guessing. That is the correct failure mode for a tenant-facing system, where a confident wrong answer about a lease or a listing can cost real money. Estate360 also scopes every query to the tenant, and that scoping is enforced by a CI guard, not by developer discipline, which is the only kind of enforcement that survives staff turnover.

Faithfulness is scored by an evaluator model against the retrieved context: does every claim in the answer trace to a chunk, or does it wander? Run it on your fixed question set in CI alongside recall. When faithfulness drops, do not just tweak the prompt; check whether the retrieved chunks actually support the question, because low faithfulness is often a retrieval problem wearing a generation costume.

Cache aggressively: exact and semantic

RAG workloads are repetitive. Cache at two levels: an exact-match cache keyed on a hash of the normalized query plus corpus version, and a semantic cache that matches near-duplicate queries at 0.85 similarity. The second catches the same question asked fifty ways. Invalidate both caches on corpus update and on embedder change. Estate360 and OnRamp both use exact plus semantic caching; it is the cheapest latency and cost win available, and it shrinks provider bills without touching quality. Measure hit rates, and remember that a stale cache serving outdated answers is worse than no cache at all, so tie cache lifetime to corpus freshness requirements.

Citations are first-class UI, not debug output

If the user cannot see which chunks produced the answer, they cannot verify it, and trust never forms. Render citations inline, link them to the source document at the right section, and make them clickable. Citations also give you free debugging: when a user reports a wrong answer, the citation trail shows whether retrieval or generation failed. Design them into the interface from the first mock, not as an afterthought bolted on after launch. Small details compound: show the document title and section, highlight the quoted span, and keep the citation count honest rather than dumping every retrieved chunk at the bottom.

Multi-format sources

Real corpora are not markdown. You will face PDFs with tables, DOCX with tracked changes, scanned images, audio call recordings, video walkthroughs. The parser layer decides your ceiling. Plan for a parser per format: text extraction with layout awareness for PDFs, structural parsing for DOCX, OCR or vision models for images, speech-to-text for audio, and frame sampling plus transcription for video. Normalize everything into the same chunk schema with source metadata (format, page, timestamp, author), so downstream stages never care where the content came from. Test parsers on ugly real files, not on the clean samples vendors show in demos. A parser that silently drops tables will cap your recall no matter how good your embeddings are.

Degrade gracefully without API keys

Development, CI and demos should not depend on live providers. Wire deterministic mocks behind the same interfaces: a fake LLM that returns templated answers from provided context, a deterministic embedder, a stub reranker. CI runs entirely on mocks plus the fixed question set's retrieval assertions, so builds never flake on provider outages or rate limits. When a real key is present, the same code paths run live. This pattern also makes onboarding new engineers painless: clone, run tests, no secrets required.

pgvector specifics worth knowing

If Postgres is your store, pgvector keeps the whole stack in one database, which simplifies backups, tenancy and transactional consistency. Create the embedding table with the tenant or namespace column you will filter on, and make that filter part of every query path, enforced in one shared function rather than copied per call site. For the keyword side, add a tsvector column with a GIN index and keep it updated with a trigger or during ingestion. Reciprocal rank fusion across the two result sets is a few lines of SQL or application code. One caveat: vector index choice (HNSW versus IVFFlat) trades build time, memory and recall; benchmark on your own corpus rather than trusting defaults, and revisit the choice whenever the corpus grows materially.

Common pitfalls I keep seeing

  • Embedding everything, citing nothing. Retrieval without citation plumbing gives you no way to debug or audit answers.
  • One index, forever. Corpora grow; indexes built on 10,000 chunks behave differently at 500,000. Re-measure recall on the fixed question set as the corpus scales.
  • Ignoring ingestion failures. A PDF that parsed to nothing does not error loudly; it just silently shrinks your effective corpus. Log and alert on per-file chunk counts.
  • Prompt tuning instead of retrieval tuning. When the answer is wrong, check the retrieved context first. Most "hallucinations" in RAG are retrieval gaps.
  • No corpus versioning. Cache keys, embeddings and eval results are all meaningless without knowing which corpus snapshot produced them.

Where RAG is the wrong tool

  • The corpus fits in context. If the whole knowledge base is a 20-page policy doc, put it in the prompt with caching and skip the retrieval stack. You will spend less engineering time and get better answers.
  • The data changes hourly. If answers must reflect stock levels or live order status, retrieval over a stale index is the wrong abstraction. Query the live system behind a tool and let the model call it.
  • The answer needs aggregation, not retrieval. "What was our average resolution time last quarter?" is a SQL question. Retrieval finds documents; it does not count rows. Give the agent a database tool instead.
  • Determinism is mandatory. Compliance checklists and financial calculations want code, not prose. Use the model to route or explain, never to compute.

A minimal rollout plan

  1. Write 20 real questions with verified answers. Build the retrieval harness first.
  2. Ingest the corpus with structure-aware chunking (380/50 as the documented start) and record the embedder version.
  3. Ship hybrid retrieval with re-ranking. Measure recall@k on the fixed set.
  4. Add faithfulness scoring. Gate it in CI next to recall.
  5. Add exact and semantic caching. Add citations to the UI. Add tenant scoping with a CI guard if you are multi-tenant.
  6. Deploy behind human review, read the failure cases weekly, and re-baseline the eval set quarterly.

RAG is unglamorous engineering: parsing, indexing, measuring, gating. The teams that treat it as such get systems users trust. The teams that treat it as "just embed the docs and chat" get demos that rot. If you want a second pair of eyes on a RAG design before you build it, get in touch; a short review often prevents a quarter of rework.