Models That Can See

Multimodal

Part of: Multimodal AI: Vision & Images

Show a photo of a fridge to a modern model and ask "what can I cook tonight?" It reads the ketchup, the half-onion, the carton of eggs, and gives you an omelette recipe. The model never "sees" pixels the way you do. It turns the image into the same kind of number-tokens it uses for text, and once that clicks, vision is easier to reason about. What it is A multimodal model is a model that accepts more than one type of input, typically text plus images, inside a single prompt. The "modes" are the input types: text, images, sometimes audio or video. The word modality just means "a kind of input." A text-only model has one modality; a vision model has two. The crucial idea: the model does not run a separate "image brain." Both your words and your picture are converted into the same internal currency, vectors of numbers, and processed by one network. Text and image meet in the same space. How it works Walk through one vision request: 1. Your image is split into patches. The picture is cut into a grid of small squares (often 14x14 or 16x16 pixels each), like tiles. 2. Each patch becomes an embedding. A vision encoder turns every tile into a vector of numbers, an image token. A 1000x1000

Challenge: The Patch Budget