Defending Against Prompt Injection

prompt injection defense

Part of: Guardrails & Moderation Filter

Your input moderation checks what the user typed. But a lot of real products don't just take user typing, they take a pasted email, a scraped webpage, an uploaded PDF, a search result, then hand that whole blob to the model as part of the prompt. Prompt injection is when that blob contains its own instructions, aimed at hijacking your system prompt, and the model, which just sees text, can't always tell your instructions from the document's. What an injection looks like The user asked for a summary. The document asked for something else entirely, and a model with weak defenses may follow the document's instructions instead, because from the model's point of view, both the system prompt and the article are just text it read. Defense 1: mark untrusted content as data, not instructions Wrap anything you didn't write yourself in explicit delimiters, and tell the model in the system prompt to treat it as data only: This doesn't make injection impossible, models can still be fooled, but it gives the model an explicit signal about which text is data, and it measurably reduces the success rate. Defense 2: heuristic phrase detection as a second layer Alongside the delimiter defense, a cheap

Challenge: Spot the Injection