Embedding Dimensions: What They Capture

Dimensions

Part of: Embeddings & Semantic Search

All module long the vectors were tiny: two or three numbers. Real embeddings are far larger. A common text model returns 1,536 floats per string , and the big ones push past 3,000. Why on earth do you need three thousand numbers to represent the word "dog"? Because each number is a separate axis of meaning , and meaning has far more than two or three sides. What it is Every slot in an embedding is a dimension , one independent direction the model can use to place a word. In the toy [action, romance] movie space you had exactly two axes, so every movie was pinned by just two facts. Real text has thousands of facts that matter at once: tense, formality, topic, sentiment, is-it-a-question, is-it-code, language, and many the model invented that have no human name. Each gets its own dimension. The headline idea: more dimensions means more room to keep different meanings apart. With two axes, "bank" the riverbank and "bank" the money place are forced near each other. With a thousand axes, the model has space to separate them along the dimensions that actually distinguish them. How it works A trained model does not assign axes by hand. It learns them, and remarkable structure falls out. S

Challenge: Dominant Axis Finder