Sending an Image to a Vision Model

image encoding for vision APIs

Part of: Caption & Alt-Text Generator

A vision model never opens a photo file. It reads bytes that have been base64-encoded into text and packed into a JSON content block next to your prompt. This lesson builds that payload. Every later lesson reuses it. What multimodal input means A normal text call sends one string. A vision call sends a list of content blocks: one block holds the image, another holds your text instruction. The model reads both as parts of the same message and reasons over them together. Turning a file into base64 A JSON request is text, so it can't carry raw binary. The image bytes get base64-encoded first, which turns them into a long string of safe characters. Building the content block The block needs three parts: a type of "image", a source dict that says how the data is encoded, and the base64 string. Sending it Why media type matters media type tells the model how to decode the bytes. The common formats are image/jpeg, image/png, image/webp, and image/gif. Label a PNG as JPEG and decoding either fails quietly or the model misreads the picture. Take this value from the file's real format. Don't guess it. The mental model to keep Every vision call is two payloads riding together: a picture trans

Challenge: Encode the Image Payload