Re-ranking & Scaling

Reranking

Part of: Vector Databases & Production Search

Your vector search returns 50 candidates in 8 milliseconds. Most are roughly right; the truly best three are buried at positions 11, 19, and 34. Re-running the whole index more carefully would be too slow. So you do the smart thing: let the fast index cast a wide net, then spend real compute re-scoring only the handful it caught. That two-stage pattern, retrieve then re-rank : is how production search gets both fast and accurate. What it is Re-ranking is a second pass that re-scores a small candidate set with a heavier, more accurate model, then reorders them. The standard architecture is two-stage retrieval : 1. Retrieve : a fast ANN search pulls a wide top-k (say 50-100 candidates). Cheap, approximate, high recall. 2. Re-rank : a slower cross-encoder scores each candidate against the query directly and reorders. Expensive, precise, run only on the survivors. The key distinction: the retriever uses a bi-encoder (embed query and docs separately, compare vectors), while the re-ranker uses a cross-encoder (feed query and doc together through a model for a relevance score). Cross-encoders are far more accurate but far too slow to run over millions of docs, which is exactly why you onl

Challenge: Retrieve-then-Rerank Pipeline