Deadlines limit how long to wait; leases limit how long an owner is recognized. Neither necessarily stops a remote process when it expires. A paused worker may wake up and finish a write after another worker has acquired the job.
The design must separate client waiting, execution, ownership, and durable outcome. This separation enables recovery without claiming that a timeout means an operation never happened.
Three clocks with different contracts
| Mechanism | Question it answers | Limit |
|---|---|---|
| Timeout or deadline | How much budget remains to wait or attempt work? | It does not prove whether a remote effect occurred. |
| Data TTL | When does this entry stop being considered current? | It does not guarantee timely notification or cancel the user of the data. |
| Ownership lease | How long is this owner recognized? | Expiry alone does not prevent late writes. |
Within a process, a monotonic clock measures duration without depending on wall-clock adjustments. Across machines, clocks are not assumed to be perfectly synchronized. The resource authority determines which expiry it accepts; a client cannot extend a hold by changing its local clock.
An operation’s budget
Depending on the library, a timeout covers connection, sending, service, and return. The application must know which phases are included. A total budget is divided among dependencies and retries, leaving time to respond or query status.
An order with three five-second attempts does not automatically fit into a ten-second request. Backoff and jitter consume budget; multiplying retries across layers can increase load precisely during an incident. Admission must also limit that work.
If the deadline expires after sending a charge, the outcome is uncertain. The operation identity is preserved, queried or retried under a compatible contract, and reconciled before opening an incompatible attempt. Idempotency protects the intended effect, while the timeout protects the wait.
Acquire, renew, and release
A coordinated acquisition records owner, expiry, and, where applicable, an ownership version. Renewal is valid only while the executor remains the owner. Release compares that identity atomically; a separate read and delete can remove the lease acquired by a successor.
The ticketing system keeps the firm hold and its expiry in PostgreSQL. Redis can support provisional coordination and maps, but losing a key does not grant a sale. The hold that confirms an operation must remain valid and belong to the authorized actor.
In etcd, a lease has a server-selected TTL and keepalive. Associated keys are deleted on expiry or revocation. The lock API returns an ownership key that can be validated within an etcd transaction. This is a concrete boundary, rather than automatic protection for any database or external provider.
The old owner can return
The diagram shows two executions around an expiry. The resource applies the ownership condition before accepting the effect.
--- config: look: handDrawn --- flowchart TD acquire["Acquire ownership<br/>Current version"] --> worker["Worker executes"] worker -->|"Pause or lost renewal"| expiry["Lease expires<br/>New owner may acquire"] expiry --> successor["Successor<br/>Later version"] worker -->|"Late write"| boundary["Resource validates<br/>ownership or fencing"] successor --> boundary boundary --> result["Accept current owner<br/>Reject old owner"]
A fencing token is a monotonic version whose comparison is performed by the protected resource, together with the necessary update in the same atomic boundary. The resource rejects versions below the latest one it has accepted. This alone does not prove that the lease remains valid or reject the old owner before a later version is observed: if the contract requires current validity, authority and deadline are also checked. A number held by the worker is insufficient if the database accepts any request. A lease ID should not be assumed to be a monotonic token either.
The condition can protect a local change and its durable result. If the effect is sending an email or calling an API that does not validate that ownership, local fencing cannot fence it off. The destination must cooperate, support compatible idempotency, or use a contract that admits reconciliation and compensation.
Recover without two valid decisions
Recovery of a pending operation checks identity, version, and known outcome. A successor does not erase history to start over; it coordinates which attempts may continue and how late responses are recorded.
In an outbox, claiming an event with a deadline may allow another publisher after a failure. A late acknowledgment from the former owner must not overwrite another attempt’s state through an unconditional update. Even with that coordination, publication can repeat, so the consumer retains deduplication.
Lease expiry does not require immediate compensation of an uncertain effect. A credit that may have completed must be resolved before returning funds as if it never happened. Cancellation and recovery decisions also need concurrency conditions.
Choose deadlines and failure behavior
The deadline considers observed latency, pauses, load, and the cost of holding resources. An overly short lease causes frequent reclaims; an overly long lease delays recovery. Renewal may also fail during a partition while the owner continues executing.
When the authority is unavailable, continuing to write based on intuition may violate the contract. Some reads can degrade with old data; a sale may require rejecting or leaving the operation pending. A single policy for every route confuses availability with safety.
Verify pauses and late responses
A worker is stopped after acquiring ownership, another is allowed to reclaim, and the first is resumed. The drill checks the condition at the resource, rather than only checking that the coordinator displays the new owner. Renewal responses and external-effect responses are also lost.
Failed renewals, reclaims, pending age, version rejections, and reconciliation duration are observed. A lease provides temporary ownership under explicit assumptions; the durable outcome and destination protection complete the contract.
References: etcd Lease API, concurrency API, and timeouts, retries, and backoff.