Handing the Model a Photo

image content blocks

Part of: Receipt Scanner

A vision model doesn't take a separate "image" parameter. It takes the same messages list you'd use for plain text. The only difference is that one block in that list is a picture instead of words. Once that clicks, the whole receipt scanner stops looking exotic: it's the chat API you already know, with a photo tucked into the same envelope as the text. What an image content block is A plain text message has content as a string. A multimodal message has content as a list of blocks , and each block carries its own type. You can put an image block and a text block in the same user turn: The source dict is where the real work happens. type says how the bytes are encoded (base64 for a raw upload). media type is the file's actual MIME type, like image/jpeg or image/png. data is the base64 string itself. Getting a photo into base64 JSON can't carry raw bytes, so an image rides across the wire as base64 text. Reading a file and encoding it is three lines: Then you call the API the same way you'd make any chat call, passing this richer content list: Why the media type has to be honest If the file is actually a PNG but you label it image/jpeg, decoding can fail or the model can misread the

Challenge: Sort the Receipt Inbox