Rate Limiting and Abuse Prevention

RateLimit

Part of: Guardrails & Safety

A startup woke up to a $40,000 bill. One script had looped a single endpoint overnight, each call asking the model for a maximum-length completion. No prompt was injected, no secret leaked, nothing was "hacked." The attacker simply used the product faster and bigger than any human could. The defense for that class of abuse is rate limiting . What it is Rate limiting caps how much a single user (or key, or IP) can consume in a window of time. For ordinary APIs you cap requests per minute . For LLM apps you usually cap tokens per window , because cost and load scale with tokens, not request count, one 100-token call and one 100,000-token call are not equal. The cap is your blast-radius control for cost abuse : token floods, retry loops, and runaway agents. Rate limiting is a different kind of guardrail than the ones earlier in this module. Injection and moderation ask "is this content dangerous?" Rate limiting asks "is this amount dangerous?" A perfectly benign request, repeated ten thousand times a minute, is an attack. How it works Two classic algorithms dominate: - Token bucket. Each user has a bucket that refills at a steady rate up to a cap. Each request spends from it; an empty

Challenge: Per-User Token Rate Limiter