Prompting With Images

Prompting

Part of: Multimodal AI: Vision & Images

Hand a model a photo of a crowded desk with no instruction and it free-associates: "A wooden desk with a laptop, a coffee mug, some papers..." Hand it the same photo with "How many cables are plugged into the laptop? Answer with just a number." and you get 3. Same pixels, wildly different answers. The image supplies the facts ; your text decides which facts come out. Learning to aim that text is the whole skill of this lesson. What it is Image prompting is combining a text instruction with an image in one request so the words steer what the model reports about the picture. The image is the evidence; the prompt is the question, the format, and the constraints. From earlier lessons you already know both the words and the picture become tokens in one shared context, so the instruction literally sits next to the image and shapes the prediction. A bare image invites a generic description. A pointed instruction turns the model into a specific tool: a counter, an extractor, a classifier, a grader. How it works Your text does three jobs at once, each pulling the answer in a direction: 1. Task : what to do: describe, count, locate, compare, judge. 2. Scope : what to look at: "only the price

Challenge: The Instruction Auditor