Content Moderation

Moderation

Part of: Guardrails & Safety

Every large AI product runs a second model you never see. Before your message reaches the main model, and often before the answer reaches you, a moderation system scores the text for harmful categories. It is the quiet immune system of production AI, and it is the layer that decides what an application will simply refuse to touch. What it is Content moderation is the automated classification of text (or images) into harm categories, hate, harassment, self-harm, sexual content, violence, illegal activity, so the system can block, flag, or route it. It is the policy enforcement layer that turns abstract rules ("no hate speech") into concrete decisions on real inputs and outputs. Moderation is distinct from the guardrails earlier in this module. Injection and jailbreak defenses ask "is someone attacking the system?" Moderation asks "is this content harmful, regardless of intent?" Both run; they answer different questions. How it works A moderation classifier outputs a score per category , usually between 0 and 1. The application sets a threshold for each category and acts when a score crosses it: Two design choices dominate: - Where to run it. Moderate the input (catch harmful request

Challenge: Moderation Eval Harness