Versioning Prompts and Models

Versioning

Part of: Deploying & Monitoring LLM Apps

Quality dropped 8 percent overnight and nobody touched the code. A teammate had edited the system prompt directly in the dashboard, and the provider had silently rolled gpt-4o from one snapshot to a newer one. Two invisible changes, zero record of either. You cannot fix what you cannot pin down, and you had pinned down nothing. The missing discipline is versioning . What it is Versioning means recording exactly which prompt and which model produced every response, so any behavior is reproducible. A prompt is not a loose string you tweak in a text box; it is an artifact with a version : a hash or a tag, that you can name, diff, and roll back to. The model is the same: gpt-4o is a moving target, but gpt-4o-2024-08-06 is a frozen one. Pin both, log both, and "why did the answer change?" becomes answerable. The rule is blunt: if you cannot say which prompt version and which model version generated a given output, you are running an experiment you cannot repeat. How it works You give each prompt a stable version derived from its content, then stamp every traced call with that version plus the pinned model id: Two ideas carry the weight. A content hash means the version changes the insta

Challenge: Release Fingerprints