How Rate Limiting Works and How to Build One

At some point in any backend system, you need to stop a single client from consuming everything. It might be a bot hammering your login endpoint, a misconfigured service firing requests in a loop, or a paying customer who went well past their plan limits. Rate limiting is how you enforce a ceiling on request volume.

Most engineers know rate limiting exists. Fewer have thought through how the algorithms actually work and why picking the wrong one creates exploitable gaps.

Fixed Window

The most intuitive approach. You define a window - say, one minute - and a limit - say, 100 requests. Each client gets a counter that resets at the start of every window.

Window: 12:00:00 - 12:01:00
Counter for client A: 0 -> 1 -> 2 -> ... -> 100 -> REJECT
Counter resets at: 12:01:00

This is fast and cheap to implement. Storing a counter per client in Redis with an expiry is a few lines of code.

The problem is boundary bursting. A client can send 100 requests at 12:00:59, wait for the window to reset, and send another 100 requests at 12:01:01. In two seconds, they have sent 200 requests. Your system sees 100 in each window and considers it fine. In practice, your backend just absorbed a 200-request spike.

For low-stakes rate limiting where boundary bursting is acceptable, fixed window is a reasonable choice. For anything where consistent load matters, it is not.

Sliding Window Log

Instead of resetting a counter on a schedule, you store the timestamp of every request and count how many fall within the last window.

Current time: 12:05:30
Window: last 60 seconds
Stored timestamps: [12:04:32, 12:04:55, 12:05:10, 12:05:28]
Count in window: 4

When a new request arrives, you remove timestamps older than the window boundary and count what remains. If the count is under the limit, the request passes and you add the new timestamp.

This is accurate. No boundary bursting because the window moves with time rather than resetting on a clock. But storing a timestamp per request for every client gets expensive at scale. A client making 1,000 requests per hour generates 1,000 stored timestamps. With millions of clients, the memory footprint becomes a real problem.

Sliding Window Counter

A practical middle ground. Instead of storing individual timestamps, you keep counters for the current and previous window, then estimate the count using a weighted average.

If the current window is 40% through and you had 60 requests last window and 20 so far this window, the estimated count is:

(60 * 0.6) + 20 = 56

This is an approximation, not an exact count. In practice it is accurate enough for most use cases, and it uses constant memory per client regardless of request volume. This is what Nginx and many API gateways use under the hood.

Token Bucket

Token bucket is the algorithm you want to understand most deeply because it is the default for API rate limiting in most production systems.

Each client has a bucket with a maximum capacity. Tokens refill at a fixed rate. Each request consumes one token. If the bucket is empty, the request is rejected.

Bucket capacity: 10 tokens
Refill rate: 2 tokens per second
Current tokens: 6

Request arrives -> tokens drop to 5 -> request allowed

The key property is that it allows bursting. A client that has been idle accumulates tokens up to the bucket capacity. When they become active, they can send a burst of requests up to that capacity before being throttled back to the refill rate. This matches real usage patterns. An API client that does batch processing sends requests in bursts, not at a perfectly steady rate.

GitHub’s API uses token bucket. Stripe uses it. Most public APIs that allow controlled bursts are running some variant of token bucket.

Leaky Bucket

Leaky bucket is the inverse mental model. Requests enter a queue (the bucket). They drain out at a constant rate regardless of how quickly they arrived. If the queue fills up, new requests are rejected.

Queue capacity: 10 requests
Drain rate: 2 requests per second

Burst of 8 requests arrives -> all queued
Processed at: 2/sec -> smooth output regardless of input burst
New request when queue full -> rejected

The output rate is constant and predictable. This is useful when you are protecting a downstream service that can only handle a steady stream. Leaky bucket smooths traffic out; token bucket allows bursts through.

The tradeoff is that leaky bucket adds latency. Requests sit in the queue before being processed. For interactive APIs where you want fast responses or fast rejections, this is often the wrong choice. For background processing pipelines or outbound traffic shaping, it is the right one.

Building One With Redis

Token bucket is the right starting point for most API rate limiters. A Redis implementation using a sorted set for the sliding window log variant:

import time
import redis

r = redis.Redis()

def is_allowed(client_id: str, limit: int, window_seconds: int) -> bool:
    now = time.time()
    window_start = now - window_seconds
    key = f"ratelimit:{client_id}"

    pipe = r.pipeline()
    # Remove timestamps outside the window
    pipe.zremrangebyscore(key, 0, window_start)
    # Count remaining requests in window
    pipe.zcard(key)
    # Add current request timestamp
    pipe.zadd(key, {str(now): now})
    # Set expiry so keys clean themselves up
    pipe.expire(key, window_seconds)
    results = pipe.execute()

    request_count = results[1]
    return request_count < limit

One subtle issue: this is not atomic. Between the count and the add, another request could sneak through and push the count over the limit. For strict correctness at high concurrency, use a Lua script to run the whole thing atomically in Redis.

The LLD Angle

If this came up in a system design interview, here is how the class structure would look. For a complete object-oriented design and Java implementation, check out our Low Level Design of a Rate Limiter.

RateLimiter (interface)
  + is_allowed(client_id: str) -> bool

TokenBucketRateLimiter (implements RateLimiter)
  - capacity: int
  - refill_rate: float
  - storage: BucketStorage

SlidingWindowRateLimiter (implements RateLimiter)
  - limit: int
  - window_seconds: int
  - storage: WindowStorage

BucketStorage (interface)
  + get_tokens(client_id) -> float
  + set_tokens(client_id, tokens, last_refill_time)

RedisBucketStorage (implements BucketStorage)
InMemoryBucketStorage (implements BucketStorage)

The interface abstracts the algorithm from the caller. The storage interface abstracts where state lives. This lets you swap Redis for an in-memory store in tests, or swap token bucket for sliding window without changing the calling code.

Where Rate Limiting Actually Gets Hard

The algorithm is the easy part. The operational challenges are where teams run into trouble.

Distributed rate limiting. If you have multiple API server instances, a per-instance counter means a client can hit 100 requests on each server separately. You need centralized state in Redis or a similar store to enforce the limit globally. This adds a Redis round-trip to every request, which needs to be fast.

Rate limiting by what. Per IP address is unreliable because shared networks (offices, NAT) look like one client. Per user ID is better but unauthenticated requests have no user ID. Per API key is the most reliable for authenticated APIs. Most production systems use a combination.

Returning the right error. A 429 response should include a Retry-After header telling the client when they can try again, and a X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset set of headers so clients can see where they stand before hitting the limit.

Burst vs sustained limits. Some APIs enforce two limits simultaneously: 10 requests per second and 1000 requests per hour. A client can burst within the per-second limit but cannot sustain that burst for the full hour. Implementing this correctly requires two separate rate limiters running in parallel.

Rate limiting looks like a small feature until you try to make it work correctly under load, across multiple servers, with meaningful error responses. The algorithm is a few lines of code. The system around it is the real design challenge.