Rate limiting · Commitwise

A search endpoint costs the company a few cents per request in a third-party API. One evening, a partner’s script with a bug calls it 2,000 times per second. Within an hour, the third-party bill is larger than the month’s budget, the database is at 100% CPU, and real users’ searches time out. Nothing was hacked; nothing was technically broken. There was just no limit.

Rate limiting caps how much traffic a client can send in a period of time. It’s one of the first things to add to any public API, and one of the most common topics in system design interviews. This lesson covers the concepts and where limits go. The Design a rate limiter lesson then builds one step by step, with distributed counters and the HTTP details.

New to some of these? Open what you need, or skip ahead.

Basics:What is an API gateway?

An API gateway is a server that sits in front of your services and handles things every request needs: authentication, routing to the right service, logging, and limits. Examples are cloud gateways, Kong, Envoy-based gateways and Spring Cloud Gateway.

Basics:What does HTTP 429 mean?

429 Too Many Requests is the HTTP status code for “you’ve sent too many requests; slow down”. It usually comes with a Retry-After header saying how many seconds to wait.

Part Not started

Why limit, where, and per what

A rate limit is like the rule at a busy bakery: “max 4 loaves per customer”. It doesn’t stop anyone from shopping; it stops one customer from emptying the shelves for everyone else.

Systems use rate limits for four reasons:

Protection: a server can handle a certain load; beyond that, it slows down for everyone.
Fairness: one heavy user shouldn’t make the service slow for all the others.
Cost: some requests cost money (SMS, AI models, paid APIs).
Abuse: guessing passwords, scraping, spamming sign-ups.

Limits can live in several places, from the outside in:

The edge (CDN or load balancer): rough limits per IP address, to stop floods before they reach your servers.
The API gateway: precise limits per user, API key or endpoint (“100 searches per minute per user”).
Inside a service: limits on especially expensive operations.
On your own outbound calls: if a payment provider allows 100 requests per second, your code must not send more.

Rate limiting is admission control by request rate per key over time. Layering matters: edge limits (CDN/WAF, per IP or ASN) absorb volumetric abuse cheaply but can’t see identity; gateway limits key on authenticated identity (user, API key, tenant) and endpoint class, and implement product tiers (“free: 60 req/min, pro: 600”); service-level limits guard specific expensive operations and dependencies; outbound limits respect third-party quotas, ideally shared across instances. Key choice is the crux: per-IP limits punish users behind carrier-grade NAT and corporate proxies (thousands of users, one IP) and are trivially evaded with IPv6 ranges, while per-user limits only work after authentication, so login and sign-up endpoints need IP plus account-based keys. Behind proxies, the client IP must come from a header set by your trusted proxy (X-Forwarded-For processed from the right, or the load balancer’s own header), never from a value the client can set.

Limits per user and endpoint at the gateway

// Spring Cloud Gateway: a Redis-backed token bucket per user, per route
@Bean
fun routes(builder: RouteLocatorBuilder, userKey: KeyResolver): RouteLocator = builder.routes()
    .route("search") { r ->
        r.path("/api/search/**").filters { f ->
            f.requestRateLimiter { c ->
                c.rateLimiter = RedisRateLimiter(10, 20, 1)   // 10/s refill, bursts up to 20
                c.keyResolver = userKey
            }
        }.uri("lb://search-service")
    }
    .build()

@Bean
fun userKey(): KeyResolver = KeyResolver { exchange ->
    exchange.getPrincipal<Principal>().map { it.name }       // authenticated user, not IP
}

// Spring Cloud Gateway: a Redis-backed token bucket per user, per route
@Bean
RouteLocator routes(RouteLocatorBuilder builder, KeyResolver userKey) {
    return builder.routes()
        .route("search", r -> r.path("/api/search/**")
            .filters(f -> f.requestRateLimiter(c -> {
                c.setRateLimiter(new RedisRateLimiter(10, 20, 1));   // 10/s refill, bursts up to 20
                c.setKeyResolver(userKey);
            }))
            .uri("lb://search-service"))
        .build();
}

@Bean
KeyResolver userKey() {
    return exchange -> exchange.getPrincipal().map(Principal::getName);   // authenticated user
}

// Express: per-user limit on search, backed by Redis so all instances share counters
import { rateLimit } from "express-rate-limit";
import { RedisStore } from "rate-limit-redis";

const searchLimiter = rateLimit({
  windowMs: 60_000,
  limit: 100,                                   // 100 searches per minute
  keyGenerator: (req) => req.user!.id,          // authenticated user, not IP
  standardHeaders: "draft-7",                   // RateLimit headers for clients
  store: new RedisStore({ sendCommand: (...args: string[]) => redis.sendCommand(args) }),
});

app.use("/api/search", requireAuth, searchLimiter, searchRouter);

Spot the bugEveryone is rate limited at once

Tap the line with the bug.

@Component
class LoginRateLimitFilter(private val limiter: Limiter) : OncePerRequestFilter() {
    override fun doFilterInternal(req: HttpServletRequest, res: HttpServletResponse, chain: FilterChain) {
        val key = "login:" + req.remoteAddr
        if (!limiter.tryAcquire(key, perMinute = 10)) {
            res.sendError(429); return
        }
        chain.doFilter(req, res)
    }
}
// The service runs behind a load balancer

@Component
public class LoginRateLimitFilter extends OncePerRequestFilter {
    @Override
    protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
            throws ServletException, IOException {
        String key = "login:" + req.getRemoteAddr();
        if (!limiter.tryAcquire(key, 10)) { res.sendError(429); return; }
        chain.doFilter(req, res);
    }
}
// The service runs behind a load balancer

const loginLimiter = rateLimit({
  windowMs: 60_000,
  limit: 10,
  keyGenerator: (req) => "login:" + req.socket.remoteAddress,
});

app.post("/login", loginLimiter, loginHandler);
// The service runs behind a load balancer

The connection’s address is the load balancer’s, not the user’s. Every login attempt from every user has the same key, so after 10 logins per minute in total, everyone gets 429. On a quiet day nobody notices; on Monday morning nobody can log in.

Take the client IP from the header your proxy sets, and only trust it from your proxy. For logins, also limit per account, so an attacker spreading guesses across many IPs is still slowed down:

// application.yml: server.forward-headers-strategy: native (or framework)
// so req.remoteAddr is the client IP taken from the trusted proxy's X-Forwarded-For
val ipKey = "login:ip:" + req.remoteAddr
val accountKey = "login:account:" + req.getParameter("email").lowercase()
if (!limiter.tryAcquire(ipKey, perMinute = 20) || !limiter.tryAcquire(accountKey, perMinute = 5)) {
    res.sendError(429); return
}

// application.yml: server.forward-headers-strategy: native (or framework)
String ipKey = "login:ip:" + req.getRemoteAddr();
String accountKey = "login:account:" + req.getParameter("email").toLowerCase();
if (!limiter.tryAcquire(ipKey, 20) || !limiter.tryAcquire(accountKey, 5)) { res.sendError(429); return; }

app.set("trust proxy", 1);   // trust exactly one hop: our load balancer; req.ip is now the client
const loginLimiter = rateLimit({
  windowMs: 60_000,
  limit: 20,
  keyGenerator: (req) => "login:ip:" + req.ip,
});
const accountLimiter = rateLimit({
  windowMs: 60_000,
  limit: 5,
  keyGenerator: (req) => "login:account:" + String(req.body.email).toLowerCase(),
});
app.post("/login", loginLimiter, accountLimiter, loginHandler);

Trusting X-Forwarded-For from anyone is the opposite mistake: an attacker sends a different fake IP in each request and is never limited.

The algorithms at a glance

Fixed window       count per clock minute            simple; up to 2× bursts at window edges
Sliding log        timestamps of every request       exact; memory grows with the limit
Sliding window     two windows, weighted             close to exact; two counters per key
Token bucket       tokens refill at a steady rate    allows bursts up to the bucket size; very common
Leaky bucket       a queue drained at a fixed rate   smooths output; adds waiting instead of rejecting

The token bucket is the one you’ll see most: each user has a bucket of, say, 20 tokens that refills at 10 tokens per second. Each request takes a token; when the bucket is empty, requests are rejected until it refills. It allows short bursts (opening an app fires several requests at once) while keeping the average rate in check. The Design a rate limiter lesson implements it.

Purposes: capacity protection, fairness, cost control, abuse mitigation.
Layers: edge (per IP/ASN, volumetric), gateway (per user/key/tenant and endpoint, product tiers), service (expensive ops), outbound (third-party quotas, shared across instances).
Keys: per-IP hurts NAT/proxy users and is evadable; per-user needs auth; login/sign-up need IP + account keys.
Client IP only from your trusted proxy (forward-headers-strategy, trust proxy hop count); never trust arbitrary X-Forwarded-For.
Algorithms: fixed window (edge bursts), sliding log/window, token bucket (bursty, common), leaky bucket (smoothing).

Check this part

Which is NOT a usual reason for rate limiting?
Limits slow heavy users down to protect everyone else.
Why does a token bucket suit mobile apps well?
Bursts up to the bucket size, steady rate on average.
Behind a load balancer, keying a limit on the connection's remote address…
Use the client IP from your trusted proxy's header.

Rate limits protect capacity, keep usage fair between clients, cap costs and slow down abuse. Put coarse limits at the edge (per IP), precise ones at the gateway (per user, API key or endpoint), and limits on your own outbound calls to third parties. Key limits on a trustworthy identity: the authenticated user or key, or the real client IP from a trusted proxy header.

Part Not started

When rate limits aren’t enough

A rate limit counts requests per second. But what overloads a server is often how many requests are in progress at the same time. If a database gets slow and each request suddenly takes 10 seconds instead of 100 milliseconds, 100 requests per second becomes 1,000 requests waiting at once, and the server runs out of threads or memory, even though the rate never changed.

So services also use:

Concurrency limits: at most N requests in progress at once (for example, 50 calls to the recommendations service). Extra ones are rejected right away, or wait briefly in a small queue.
Bulkheads: separate limits for separate dependencies, so a slow recommendations service can’t use up the threads that payments need. (The name comes from ships, which are split into watertight compartments.)
Load shedding: when the service is overloaded, reject the least important requests first (prefetching, analytics) to keep the important ones (checkout) working.

By Little’s law, in-flight requests = arrival rate × latency; a dependency’s latency spike multiplies concurrency at a constant rate, so concurrency limits (semaphores, bulkheads in Resilience4j, Envoy’s circuit-breaker thresholds) protect resources that rate limits can’t. Adaptive concurrency limits (Netflix’s concurrency-limits library, Envoy adaptive concurrency) adjust the limit from observed latency, à la TCP congestion control. Load shedding prioritizes by request class (critical, user-facing, background), often using queue time or CPU as the signal, and rejects early with a cheap response rather than timing out late. Combine with timeouts shorter than the caller’s, circuit breakers that stop calling a failing dependency, and clients that retry with backoff and a retry budget so recovery isn’t swamped. The implementation detail that bites: every permit must be released on every path, including exceptions, cancellations and timeouts.

A bulkhead around a slow dependency

// Kotlin coroutines: at most 50 concurrent calls to recommendations
class RecommendationsClient(private val api: RecommendationsApi) {
    private val permits = Semaphore(50)

    suspend fun forUser(id: UserId): List<Product> {
        if (!permits.tryAcquire()) return emptyList()          // shed: recommendations are optional
        try {
            return withTimeout(300) { api.recommendations(id) }
        } catch (e: TimeoutCancellationException) {
            return emptyList()
        } finally {
            permits.release()                                  // on every path
        }
    }
}

// Resilience4j: a bulkhead plus a time limit, with a fallback
Bulkhead bulkhead = Bulkhead.of("recommendations", BulkheadConfig.custom()
        .maxConcurrentCalls(50)
        .maxWaitDuration(Duration.ZERO)                        // reject immediately when full
        .build());

public List<Product> forUser(UserId id) {
    Supplier<List<Product>> call = Bulkhead.decorateSupplier(bulkhead, () -> api.recommendations(id));
    return Try.ofSupplier(call)
            .recover(BulkheadFullException.class, e -> List.of())   // shed: recommendations are optional
            .get();
}

// A tiny semaphore for outbound calls
class Semaphore {
  private inUse = 0;
  constructor(private readonly max: number) {}
  tryAcquire(): boolean {
    if (this.inUse >= this.max) return false;
    this.inUse++;
    return true;
  }
  release() { this.inUse--; }
}

const permits = new Semaphore(50);

async function recommendationsFor(id: string): Promise<Product[]> {
  if (!permits.tryAcquire()) return [];                       // shed: recommendations are optional
  try {
    return await api.recommendations(id, { signal: AbortSignal.timeout(300) });
  } catch {
    return [];
  } finally {
    permits.release();                                        // on every path
  }
}

Spot the bugAfter a bad hour, every request is rejected

Tap the line with the bug.

suspend fun forUser(id: UserId): List<Product> {
    if (!permits.tryAcquire()) return emptyList()
    val result = withTimeout(300) { api.recommendations(id) }
    permits.release()
    return result
}

public List<Product> forUser(UserId id) {
    if (!permits.tryAcquire()) return List.of();
    List<Product> result = api.recommendations(id);   // throws on timeout
    permits.release();
    return result;
}

async function recommendationsFor(id: string): Promise<Product[]> {
  if (!permits.tryAcquire()) return [];
  const result = await api.recommendations(id, { signal: AbortSignal.timeout(300) });
  permits.release();
  return result;
}

Production puzzles

Predict the output

A payment provider allows 100 requests per second per account. The service calling it runs on 8 instances, each with a local limiter of 100 per second. What happens on a busy day?

Predict the output

A service is overloaded. It queues incoming requests in an unbounded queue and processes them in order; clients time out after 2 seconds. Queue wait reaches 5 seconds. What fraction of the work it does is useful?

Choosing the response

429 Too Many Requests: this client exceeded its limit. Include Retry-After and RateLimit headers.
503 Service Unavailable: the service is overloaded for everyone (load shedding). Include Retry-After if you can.
A degraded response: for optional data (recommendations, avatars), return something smaller instead of an error.

Exerciseeasy

Protect a sign-up endpoint

A sign-up endpoint sends a verification SMS, which costs money. Bots have started creating thousands of accounts. Suggest limits: what keys, what numbers, and what else you’d add.

›

There’s no logged-in user yet. What do you know about the request? And what does each sign-up cost?

SolutionTry it yourself first

Per IP (from the trusted proxy): for example 5 sign-ups per hour. It slows simple bots but not distributed ones, and must be generous enough for shared networks.
Per phone number: 3 SMS per day. This caps the cost per target and stops “SMS bombing” a victim’s number.
Per phone number prefix or country: an alert or a stricter limit when one country suddenly spikes, which is a common sign of SMS fraud.
A global budget: a maximum number of SMS per hour for the whole service, so a bug or an attack can’t produce an unlimited bill.
Beyond limits: a CAPTCHA or device attestation (App Attest, Play Integrity) for mobile sign-ups.

Little’s law: in-flight = rate × latency; latency spikes overload at a constant rate, so cap concurrency (semaphores, bulkheads, adaptive limits).
Load shedding by priority and queue time; bounded queues; reject early with 503 + Retry-After instead of timing out late.
Outbound quotas need shared limiters (Redis, egress proxy) across instances; still honor upstream 429s.
Release permits on every path (finally, withPermit, library decorators); leaked permits turn a slow hour into a permanent outage.
Pair with timeouts, circuit breakers and client retry budgets.

Check this part

What does a concurrency limit cap?
Slow requests pile up even when the rate is constant.
Eight instances each limit outbound calls to 100/s, and the provider allows 100/s per account. Result?
Account-wide quotas need a shared limiter.
Why release a semaphore permit in finally?
Leaks outlive the incident that caused them.

A rate limit doesn't protect against slow requests piling up: concurrency limits and bulkheads cap work in progress, load shedding drops low-priority work when a service is saturated, and outbound limiters respect other services' quotas. Every permit taken must be released, including on errors and timeouts.

Key takeaways

Rate limits protect servers, keep usage fair, control costs and slow abuse.
Limit at the edge, at the gateway, and on your own calls to other services.
Key limits on the user or on the real client IP from your proxy.
Concurrency limits and load shedding protect a service when requests get slow.

Layered limits keyed on trustworthy identity; trusted-proxy IP extraction.
Token bucket for bursty clients; know the trade-offs of the others.
Concurrency limits, bulkheads, load shedding and shared outbound limiters.
Release permits on every path; reject early rather than time out late.