Useful capacity is work that finishes within the latency and correctness contract. Accepting every request and responding after the client has abandoned it can consume more resources without providing more service. Backpressure communicates pressure upstream; admission decides which work can enter.
The starting point is to measure the bottleneck and distinguish reads, holds, charges, and background processing. A single RPS figure does not describe their costs or priorities.
A budget with units
Consider a hypothetical ticketing peak: twelve thousand catalog reads and two thousand hold requests per second. This is not any provider’s capacity. If a stable flow of five hundred holds per second is admitted and each takes an average of two hundred milliseconds within a measured boundary, about one hundred holds will be in progress within that boundary.
The calculation is rate times duration: 500/s × 0.2s = 100. This relationship is useful for averages in a stable system with well-defined boundaries. It does not turn p99 into exact concurrency or demonstrate that the database supports that load. A benchmark with realistic data distribution, locks, writes, and failures validates the budget.
Capacity is prepared with margin before the peak. Instance startup, cache warming, and connection establishment also take time.
Admit before saturation
A concurrent-work or rate limit respects dependency capacity. The policy can allocate budgets by route and actor, prioritize confirmations over browsing, and reserve capacity for recovery. Rejecting early may preserve more useful work than accumulating unbounded waits.
A Rate Limiter controls one dimension of consumption; it does not necessarily replace concurrency-based admission. A long task and a short one may use the same request count while imposing different costs.
The response communicates whether the client may retry and under which contract. It does not offer an exact deadline when the server does not know one. Clients and SDKs must respect the budget rather than turn rejection into an immediate surge of new attempts.
Finite queues and work age
A queue absorbs a temporary difference between arrivals and completions. If arrivals continuously exceed departures, it grows until it reaches a limit or loses usefulness. Message count, bytes, and age represent different constraints.
As a hypothetical example, three thousand tasks arriving per second and two thousand completing for sixty seconds add sixty thousand tasks. If arrivals then fall to one thousand per second while completion remains at two thousand, ideal draining takes another sixty seconds. Retries, variable costs, and pauses change that calculation.
Message Queues needs capacity, expiry, duplicate, and recovery policies. A queue does not guarantee that a hold remains valid when processed. Its authority and deadline are checked again before applying the effect.
An explicit pressure boundary
The diagram separates waiting from granting an operation. The virtual queue represents admission and does not own inventory.
--- config: look: handDrawn --- flowchart TD incoming["Request burst"] --> admission["Admission<br/>Budget by route and actor"] admission -->|"Capacity available"| workers["Bounded concurrent work"] admission -->|"Acceptable wait"| queue["Finite queue<br/>Age and deadline"] admission -->|"Budget exhausted"| reject["Early rejection<br/>Bounded retry"] queue --> workers workers --> database["Authoritative database<br/>Measured capacity"] database -->|"Latency, load, and errors"| admission
Load signals can reduce admission, but the controller also needs stability: thresholds, windows, and gradual recovery appropriate to the measurement. Oscillating between opening everything and closing everything transfers instability to the client.
Connections are not SQL capacity
The budget considers the sum of pools across all instances. Ten pods with a maximum of twenty connections can request two hundred, in addition to workers, administration, and other services. These are the example’s configured maxima; they do not justify creating that many without measuring activity and engine limits.
A pool reuses connections and can bound waiting. RDS Proxy can pool connections, queue or reject according to limits, but it does not add CPU or I/O or remove PostgreSQL lock contention. Session state and pinning can reduce effective reuse.
Adding pods or pools while the database is already saturated may worsen the problem. Queries, indexes, and short transactions are reviewed before distributing more load. Sharding is another decision with costs, rather than an automatic response to every burst.
Failures, retries, and fairness
A timeout does not immediately release all remote work. If every layer retries, one request can become many operations. Backoff with jitter, attempt limits, and one responsible retry layer where feasible reduce amplification without replacing idempotency.
When Redis or a cache fails, an unbounded fallback can send the entire peak to the database. Caching requires a degradation budget. Some reads tolerate old data; a firm hold is still checked by its authority.
Fairness needs an identity and a policy: actor limits, protection against monopolization, and capacity for new users or recovery. FIFO does not demonstrate fairness if one producer occupies almost the entire queue.
Verify useful work
The test reproduces bursts, a slow dependency, a cold cache, and concurrent retries. It measures operations completed in time, rejections, queue age, active connections, contention, and p95/p99 latency by route.
The ticketing system must preserve one valid sale even when rejecting or delaying work. A good outcome preserves invariants and observable recovery; a high-RPS graph with expired clients does not demonstrate useful availability.
References: load shedding to avoid overload, retry budgets, and RDS Proxy capabilities.