An API can respond quickly while duplicating an order; a process can be healthy while leaving events pending. Observability explains behavior through signals. Verification checks whether that behavior preserves a contract.
Both start with a concrete question: which result must exist, which anomaly matters, and at which boundary is it decided? Adding logs to every method does not replace that definition.
Invariants before the dashboard
In ticketing, one invariant is that a seat and event do not have two valid sales. An order must also retain its payment attempts, and an uncertain charge needs a recovery path. These rules are enforced at the corresponding authority; an alert reports a deviation but does not prevent it.
For a projection, the contract may require that an entity’s version never moves backward and that missing events are recovered. For a worker, the outcome may be finished, recoverably pending, or definitively failed with a cause. Measuring only HTTP 200 conflates acceptance and completion.
| Contract | Useful signal | Verification |
|---|---|---|
| One valid sale per seat | Conflicts and durable decisions. | Query authoritative state under concurrency. |
| Recoverable uncertain attempts | Pending count and age. | Lose responses and reconcile by identity. |
| Projection within its freshness contract | Lag, version, and stalled events. | Compare with the source without assuming the consumer is current. |
| Useful work within the deadline | Latency, rejections, and queue age. | Introduce load and a slow dependency. |
Identities for following the outcome
A request ID identifies a request; an operation ID identifies the logical operation and remains stable across its retries; an event ID identifies the logical event and is preserved across repeated publications. One operation can have multiple requests and deliveries. Correlating them avoids interpreting every retry as a separate purchase or every span as an acknowledged effect.
Trace context links spans across services when propagated correctly. OpenTelemetry also organizes metrics and logs, but an incomplete trace does not prove that no effect occurred. Sampling, crashed processes, and export failures may hide part of the flow.
Context is validated at trust boundaries, and secrets or personal information are kept out of baggage. Identities are retained in the business record when recovery needs them; the telemetry system does not replace that durable record.
Service and business signals
Latency, traffic, errors, and saturation describe the service but need units and boundaries. Queue time, execution, dependencies, and response are distinguished. A client timeout may coexist with a successful commit.
For capacity, p95/p99 by operation and work completed before its deadline matter. For recovery, pending age, retries, discrepancies, and time to resolution matter. An SLI specifies population, window, and success condition; an SLO sets a target for that indicator.
An error budget guides decisions but does not authorize violations of financial invariants. A catalog may tolerate old responses within its contract; a double sale requires a different protection from permitting a percentage of high latency.
An evidence loop
Signals and drills compare outcomes with the contract. Reconciliation queries the authority rather than deciding based on the absence of a log.
--- config: look: handDrawn --- flowchart TD operation["Operation<br/>Stable identity"] --> authority["Durable decision<br/>Invariants"] operation --> signals["Metrics, logs, and traces<br/>Controlled context"] authority --> verify["Verify outcome<br/>and reconcile"] signals --> verify tests["Concurrency and<br/>failure drills"] --> verify verify --> action["Correct, recover,<br/>or adjust contract"]
A report distinguishes observed, inferred, and unverified information. A span shows an instrumented action; it does not independently certify complete causality or atomicity across stores. Claims about consistency and authority require evidence at those boundaries.
Cardinality, cost, and access
Every combination of Prometheus labels creates a time series. Normalized route, result class, and service are dimensions that can be bounded. A user ID, email, or request ID for every request may cause unbounded growth and expose data.
Detailed identifiers can be useful in controlled logs or traces with appropriate permissions, retention, and redaction. Copying complete payloads is unnecessary to explain a decision. Signal export, storage, and query costs are also designed.
The telemetry pipeline has its own buffers and failures. If it stops, the application needs defined behavior: bound memory and assess which evidence is lost. Blocking a sale indefinitely because an exporter is down can create another incident.
Test the difficult windows
Idempotency tests lose the response after commit, send concurrent duplicate keys, and receive late outcomes. Outbox tests interrupt the window between publication and checkpoint, duplicate events, and rebuild a projection.
Recovery tests isolate a writer, promote a replica, and restore a backup. Admission tests overload a dependency without permitting an infinite queue. Every drill declares initial conditions, the injected failure, success criteria, and test-environment cleanup.
Checking that a process restarted is insufficient. Outcomes, uniqueness, pending states, and their recovery are queried. External effects use safe environments and contracts, rather than submitting a real charge merely to test a timeout.
Turn evidence into operation
A useful alert identifies the threatened contract, affected population, and where to investigate. A runbook connects signals to verifiable recovery decisions: query authority, pause admission, reconcile an attempt, or restore a rehearsed copy.
Discrepancies are recorded without erasing history. After recovery, backlog drainage is verified, and uncertain states have an owner and deadline. Observability provides explanation; constraints, protocols, and drills sustain correctness.
References: OpenTelemetry context propagation, signals, and Prometheus names and labels.