Overfitting and Generalization

Overfitting

Part of: Fine-Tuning & Evals

Your fine-tune scores 99% on the data you trained it on. You ship it. In production it scores 71%. Nothing broke, you just measured the wrong thing. A model that aces the exact questions it studied has told you almost nothing about how it handles new ones. That gap between studied and unseen is the single most important idea in evaluating any trained model. What it is Overfitting is when a model learns the training examples too well, memorizing their specific quirks instead of the general rule. It looks brilliant on data it has seen and falls apart on data it hasn't. Generalization is the opposite and the actual goal: performing well on inputs the model was never trained on, because it learned the underlying pattern rather than the answer key. The only way to tell them apart is to never let the model see its own exam. You split your data: - Training set : the model learns from these. - Test set (held-out) : locked in a vault, never used in training, used only to measure real performance. How it works You hold back a slice of data before training, commonly 80% train, 20% test, and you guard it. The model trains on the 80%, then you score it on the untouched 20%. The comparison is th

Challenge: The Overfit Detector