A/B Testing LLM Changes

ABTest

Part of: Deploying & Monitoring LLM Apps

A team was certain their new prompt was better. The eval suite agreed, the demo looked sharper, everyone shipped it to 100 percent of traffic. A week later the thumbs-up rate had quietly fallen and support tickets ticked up. Offline scores said one thing; real users said another. The fix is not a better opinion, it is an A/B test that lets live traffic decide. What it is An A/B test splits live traffic between two versions, a control (A, the current prompt or model) and a variant (B, the proposed change), and compares them on a real metric. Instead of arguing about which is better, you measure: send half the users to A, half to B, and watch a number that matters, like the rate of helpful answers, successful task completions, or thumbs-up. The key word is live . An eval (lesson 3) measures quality on a frozen set of cases; an A/B test measures it on the messy distribution of real requests, including the ones you never thought to put in your eval. The two are complementary: evals gate the merge, A/B tests gate the rollout. How it works You assign each request deterministically to a bucket, run the matching version, and tally outcomes per bucket: Two details make this trustworthy. The

Challenge: The Experiment Readout