Calling a Multimodal Vision Model

Multimodal API

Part of: Computer Vision Basics

Five years ago, "make the computer tell me what's in this photo" meant collecting thousands of labeled images, training a CNN, and deploying it, weeks of work for one narrow task. Today it's one API call. You send the image to a multimodal model like Claude and ask in plain English: "What's in this picture?" "Is the person wearing a helmet?" "Read the receipt total." It answers in words. That collapse from weeks to minutes is the most important shift in everyday vision work. What "multimodal" means A multimodal model accepts more than one kind of input. Claude takes text and images in the same message . Under the hood it still converts the image into numbers and runs them through learned vision layers, the CNN ideas from the last lesson didn't disappear, they got absorbed into a much larger model. What changed for you is the interface: you talk to it like a person instead of training a custom classifier for every new task. The shape of the request You send a message whose content is a list of blocks : typically one image block and one text block. The image travels as base64-encoded bytes tagged with its media type (e.g. image/png). Base64 turns raw binary into text characters so th

Challenge: Triage the Vision Model's Verdicts