When Retrieval Fails

Fallback

Part of: Build a RAG System

Someone asks your document Q&A tool, "What's the company's policy on Mars colonization?" There is no such policy. Retrieval dutifully returns the three least irrelevant chunks it could find, and your model, handed garbage context, confidently summarizes the garbage. The system didn't break loudly. It failed silently, which is worse. What it is Retrieval failure is when the vector store has nothing genuinely relevant, yet still returns something , because similarity search always returns the top-k, even when the top match is weak. The fallback is the deliberate decision to refuse: detect that the best match is too weak, and have the system say "I don't know" or "not in the docs" instead of answering from junk. Top-k retrieval never returns empty. Ask about Mars and it hands back your three closest chunks no matter how far away they are. The closest chunk to an unanswerable question is still a wrong chunk. How it works The fix is a similarity threshold . After scoring, look at the best chunk's similarity. If even the best score falls below a cutoff, the store has nothing relevant, so refuse before you ever build a prompt. Two layers protect you, and you want both: 1. Retrieval-side g

Challenge: The Refusal Gate