Weights and Parameters

Weights

Part of: How AI Actually Works

"GPT-3 has 175 billion parameters." You've seen numbers like this thrown around like horsepower stats. But what is a parameter? It is just a single number. The whole model, everything it knows about grammar, facts, code, and tone, is stored in billions of these plain numbers, and nothing else. What it is A parameter (also called a weight ) is one tunable number inside the model. Lesson 3 mentioned weights as the thing training adjusts; here we open the box. A model is a giant grid of these numbers organized into layers. When you run a prompt through, the model multiplies your token values by these weights, adds them up, and passes the result forward, over and over, through every layer, until logits pop out the other end. There is no separate "knowledge database." The fact that Paris is the capital of France is not stored as a sentence anywhere. It is smeared across millions of weights as a pattern that makes "Paris" the likely continuation of "The capital of France is." How it works Each weight starts as a random number. Training (lesson 3) nudges every weight, billions of times, until the whole grid predicts text well. A toy version of the multiply-and-sum that weights perform: A

Challenge: Parameter Budget Report