Checking the Output Too

output moderation

Part of: Guardrails & Moderation Filter

A clean, friendly input can still produce a bad output. A clever jailbreak might walk the model past its own guidelines. It might hallucinate a fake but convincing piece of personal data. Input moderation catches none of this, because the problem shows up after the input already looked fine. That's why a serious guardrails system checks both directions . The two gates Input moderation decides whether to make the call at all. Output moderation decides whether the reply that came back is safe to actually show. They use the same classifier machinery, just pointed at a different piece of text. Block isn't your only option Blocking an entire reply because one sentence in the middle mentioned something sensitive throws away a lot of otherwise-fine content. A softer response for lower-severity findings is redaction : replace the offending span with a placeholder and let the rest through. re.escape matters here: a term with special regex characters (a period, a plus sign) would otherwise be interpreted as regex syntax instead of a literal string. The \b word boundaries keep this from mangling substrings, the same lesson from the blocklist. Matching the response to the severity A useful sha

Challenge: Redact Sensitive Spans