A load balancer selects a destination for a connection or request. It distributes traffic across instances and removes destinations that cannot serve it, while introducing its own capacity, configuration, and failure modes. The goal is to preserve a useful service as load and available servers change.
The design starts with the unit being distributed, the work required to serve it, and the behavior allowed when destinations change. Two requests can have very different costs even if each counts as one.
L4 and L7
An L4 load balancer makes decisions using transport information such as addresses, ports, and TCP flows. It typically assigns a connection or flow to a destination; requests traveling over a persistent connection may stay together. Distributing connections does not imply distributing work evenly.
An L7 load balancer understands an application protocol, such as HTTP, and can route by host, path, or headers. It may terminate TLS and select destinations per request when the product and protocol allow it. This adds processing and requires deciding where traffic is encrypted again and which headers are trusted. Exact capabilities depend on the selected service.
This decision connects to the API contract: timeouts, size limits, streams, and long responses affect how long a request occupies resources. WebSockets, persistent connections, and streams need an explicit policy during deployments.
Selection and affinity
Round robin distributes turns. It works better when destinations and their requests have comparable costs. Weighted round robin represents different capacities, but weights need maintenance. Least connections favors destinations with fewer connections; an idle connection and one performing expensive work count equally if the signal does not distinguish their cost.
Variants that consider pending requests or latency try to reflect load more closely. They need verification with skewed distributions and slow responses: a delayed signal may send too much work to a seemingly free destination. No algorithm creates capacity when every destination is saturated.
Affinity keeps some requests on the same destination through cookies or hashes. It can help a migration or exploit local caches, but concentrates active users and makes an instance failure affect its sessions. If required state lives only in local memory, rebalancing can lose it. Shared state with a recovery contract reduces that dependence; it also adds a shared dependency.
Health checks and the eligible pool
Checks should distinguish a live process from an instance ready to receive traffic. A quick HTTP response on a health route does not prove that the main route maintains latency under load. Thresholds, intervals, and timeouts determine how long failure detection takes and when an instance returns to the pool.
--- config: look: handDrawn --- flowchart TD client["Client"] --> admission["Admission and request deadline"] admission --> balance["Load balancer<br/>L4 flow or L7 request"] checks["Health checks"] -->|"Update eligible pool"| balance balance -->|"New traffic"| healthy["Healthy instances"] balance -.->|"Stop new traffic"| draining["Draining instance"] draining -->|"Finish bounded in-flight work"| retire["Retire instance"] healthy --> data[("Shared authoritative state")] draining --> data
A check that queries every dependency from every instance can amplify a shared outage. If all instances declare themselves unhealthy because the database is slow, removing them does not repair the database. Define which failures justify removing an instance and which require degradation or admission limits. Review the selected product’s policy when every destination fails its check: it may reject requests or try to continue sending traffic, with different consequences. ALB can fail open when all targets in a group are unhealthy across every enabled AZ, sending traffic to them. The diagram shows the usual path with healthy destinations, rather than a guarantee that all traffic avoids unhealthy ones.
A new instance needs capacity ready and, where applicable, warmed connections or caches before receiving its full share. Marking it healthy while it starts expensive work shifts the problem to its first requests.
Overload and retries
Load balancing needs bounded queues, deadlines, and admission limits. Letting a queue grow indefinitely increases wait time and consumes resources even after clients abandon their requests. Relevant signals include active and pending work per destination as well as requests per second.
A timeout may occur after the destination commits a write. Forwarding the request to another instance does not reveal whether the first one finished. Retryable operations need idempotency and the same operation identifier, preserved across retries according to their contract. A load balancer does not make a distributed operation atomic or guarantee exactly-once execution.
Retries need budgets, backoff, and jitter. If the client, gateway, and application each retry several times, they multiply load during an incident. They must also preserve the total deadline: starting another request without enough remaining time consumes capacity with little chance of delivering a useful result.
Failover and draining
Availability requires redundancy for the load balancer and destinations in suitable failure domains. Verify what happens when a zone is lost: remaining traffic may exceed survivor capacity. DNS, client caches, and existing connections can delay a change; adding a second address does not create instantaneous failover across every path.
Draining removes a destination from selection for new traffic and lets in-flight work finish within a defined window. Deployment must coordinate deregistration, the process termination signal, and its shutdown deadline. When that deadline expires, a long connection may be cut; a write may leave an uncertain outcome for the client. A queue consumed by the instance also needs its own acknowledgements and idempotency.
What to verify
A useful test combines uneven load, a slow destination, zone loss, and deployment with persistent connections. Make a timeout coincide with a committed write and verify the retry. During draining, observe new traffic, pending work, and shutdown when the deadline expires.
Signals include full-request p95/p99, errors and rejections, work distribution across instances, queues, active connections, and detection and recovery time. The conclusion should state remaining capacity after a failure and which operations may have uncertain outcomes. Distributing destinations improves operation when these limits are also defined.