Monitoring Search Quality

Monitoring

Part of: Vector Databases & Production Search

Search quality does not fail with an error. It rots quietly: a new embedding model ships, the corpus drifts, an index is rebuilt with a lower setting, and one Tuesday users start getting subtly worse answers while every dashboard stays green. Nothing crashed. Recall just slid from 0.95 to 0.82, and no one noticed for three weeks. Monitoring search quality is how you catch the silent rot before your users do. What it is Monitoring search quality means continuously measuring whether retrieval is still good, instead of assuming it. The three signals that matter most are recall drift (is the index still finding the right documents?), latency (especially p95/p99 tail latency, not just the average), and index freshness (how stale is the indexed data versus the source?). You compare each against a baseline and alert when it crosses a threshold. How it works Track each metric per time window and fire an alert when it breaches a bar. Recall drift is usually expressed as a percentage drop below a baseline: You measure recall by running a fixed set of labeled queries (a golden set) against the live index and comparing the returned neighbors to known-good answers - the same recall@k from earli

Challenge: Search Quality Alerter