A rate limiter decides how much traffic a system can admit. It protects capacity, shares resources among clients, and enforces quotas. When multiple instances receive requests from the same client, they must coordinate consumption of the budget they share.
The design can be explored in five steps: define requirements, establish the interface, choose the algorithm and its placement, coordinate state, and resolve operational trade-offs.
Requirements
The functional requirements are to identify the client, select its rules, and decide whether a request can proceed. Rules must be configurable by user, application, tenant, or endpoint. When the client exhausts its budget, the API responds with HTTP 429 Too Many Requests and can include Retry-After to indicate how long it should wait.
Identity must come from trustworthy information, such as the authenticated user or a validated API key. An IP address can support certain limits, but a shared network may group many users under the same address.
The nonfunctional requirements are low latency, high availability, fault tolerance, and an aggregate target of one million requests per second. That figure is a capacity target that must be validated against the actual distribution of clients, rules, and traffic. The design must also define how much excess traffic is acceptable during a failure and how much additional latency the decision can introduce.
Entities and interface
The central entities are the identified client, the applicable rule, and the request that consumes budget. The limiter’s state records how many tokens remain and when their refill was last calculated.
A conceptual interface can be expressed as:
allow(subject, rule, cost) -> { allowed, retryAfter }subject identifies who the limit applies to; rule defines the capacity and refill rate; cost specifies how many tokens the operation consumes. A simple request can cost one token, while a more expensive operation can consume several.
Reading the budget, making the decision, and updating state must form a single atomic operation. If two gateways read the same balance and update it separately, both could approve work using the same tokens.
High-level design
The limiter sits at the edge or in the API gateway, before work reaches the service. This allows traffic to be rejected early and a common policy to be applied across multiple services. Gateway instances share state to coordinate requests from the same client.
--- config: look: handDrawn --- flowchart TD client["Authenticated client"] --> gateway["API gateways<br/>Instances 1 to N"] gateway -->|"Check and consume budget"| redis[("Redis primary<br/>Token Bucket + Lua")] redis -->|"allowed + retryAfter"| gateway gateway -->|"Request admitted"| service["Protected service"] gateway -->|"Budget exhausted"| rejected["HTTP 429<br/>Retry-After"] rejected --> client redis -.->|"Asynchronous replication"| replica[("Replica for failover")]
Each bucket belongs to a primary. Gateways coordinate its consumption there; the replica allows the service to recover after a failure and may lag behind the primary’s state.
The chosen algorithm is Token Bucket. Each bucket has a maximum capacity B and refills tokens at a rate r. Before deciding, it calculates the available balance:
tokens = min(B, previousTokens + r * elapsedSeconds)If enough tokens are available, the limiter deducts the cost and allows the request. Otherwise, it rejects it. The estimated time until sufficient budget is available is:
waitSeconds = (cost - tokens) / rThis expression assumes r > 0 and a cost that fits within the bucket. For a Retry-After expressed in whole seconds, the wait is rounded up. Other requests may consume tokens that refill during that interval.
For example, a bucket with a capacity of 100 tokens and a refill rate of 10 tokens per second can admit an initial burst of up to 100 requests costing one token each. Once exhausted, it recovers budget at a rate of 10 per second. Capacity controls bursts and refill controls the sustained rate; the algorithm permits bursts and can exceed 10 requests within a particular second.
Shared state in Redis
Redis keeps state in memory and allows gateways to consult the same bucket. Each decision runs a Lua script that reads the balance, calculates the refill, checks the cost, and stores the resulting state.
Redis executes the script atomically with respect to other operations on that server. In Redis Cluster, all keys used by a script must belong to the same hash slot. Keeping each bucket’s state in a single key simplifies that constraint. A rule that combines budgets from multiple keys must explicitly resolve where they are coordinated.
Refill also requires a consistent time reference. It is advisable to calculate it using the clock of the server that owns the bucket and guard against backward clock jumps. Keys can expire after an inactivity period long enough for the bucket to refill completely; deleting them sooner could restore budget prematurely.
Scripts must be short: while running, they occupy the server and can increase the latency of other decisions. Bucket state needs a memory policy that prevents silent eviction, because losing a bucket can reset its budget.
Scalability
Redis Cluster distributes buckets across shards. This spreads decisions for many clients and allows capacity to grow horizontally. The number of shards must be selected using measured performance per decision, including Lua, networking, state size, and operational headroom.
A very active client can concentrate traffic on a single key and saturate its shard. That key still has a single owner even if more shards are added. Such cases can combine local limits with a global budget, or allocate portions of the budget to gateways. Both strategies must define how budget is distributed and recovered, and what deviation is acceptable.
Availability and failure behavior
Replicas allow the service to recover if a primary fails. With asynchronous replication, a promotion can lose recent bucket updates and admit traffic beyond the intended limit. Decisions that consume tokens must go through the primary; a replica with stale reads can report an incorrect balance.
The design must also decide what happens when the gateway loses access to the limiter:
- Fail open: allow requests to continue. This favors availability and can increase load on the protected services.
- Fail closed: block the operation. This protects the budget, but makes service availability depend on the limiter.
- Emergency local limit: admit a bounded volume per gateway while coordination recovers. Aggregate excess traffic depends on the number of active gateways and the local budget each receives.
The policy can vary by endpoint. An expensive or sensitive operation may require different behavior from a public read. HTTP 429 corresponds to an exhausted quota; if the system blocks because it cannot consult the limiter, HTTP 503 may better communicate temporary unavailability.
Latency and geographic distribution
Connection pooling reuses Redis connections and avoids establishing one for every request. Timeouts bound how long the gateway waits before applying its failure policy. Latency must be measured at percentiles such as p95 and p99, including the network hop and the wait to execute the script.
A timeout leaves the result uncertain: Redis may have deducted tokens even if its response never reaches the gateway. Repeating the decision can deduct tokens twice for the same attempt. The retry policy must account for that case; if deduplication is required, the design must identify each attempt and remember its decision.
Moving state closer to the user reduces latency, but requires deciding the scope of the quota. A strict global budget needs coordinated consumption across regions. Independent regional buckets reduce that coordination; their combined budget must be partitioned to preserve a bounded global quota.
The choice depends on what matters for that product: global accuracy, the ability to keep operating during a partition, or decision speed. Redis Cluster partitions state across shards, but coordinating quotas across regions requires an additional design.
What to verify
The design must be tested with concurrent requests for the same client, bursts, clients that produce hot keys, and primary changes. It is also useful to simulate lost connectivity to Redis and observe admitted traffic and gateway latency.
The central signals are admitted and rejected requests, errors consulting the limiter, decision time, and activations of the emergency policy. Together, they help evaluate whether the system protects the budget and maintains performance under the conditions it was designed for.