Input Moderation with a Classifier

classifier-based input moderation

Part of: Guardrails & Moderation Filter

The blocklist catches vocabulary. It cannot catch intent phrased in ordinary words: "how would someone get past a building's badge readers without being seen" contains not a single banned term but is clearly asking for help with something you don't want your app assisting with. For that, you need a classifier call : a second, cheap model call whose only job is to label the input before your main call ever runs. The classification prompt Ask for a fixed, small set of boolean categories, and say explicitly that no prose or fences are wanted: A pinned, fixed schema (the same four keys, every time) is what makes this parseable. "Return any categories you notice" invites a different shape every call, and your parser breaks the first time the model gets creative. Parsing defensively Models still occasionally wrap "pure JSON" in a sentence or a code fence. Extract the object the same way you'd extract any structured reply, from the first { to the last }: Turning categories into a decision The classifier's job stops at labeling. Deciding what to do with the labels is separate (you'll formalize that as a policy config in lesson 6), but the simplest version is: any flagged category means don

Challenge: Parse the Classifier Verdict