Design a Health Check / Status System
How to design health checks for interviews: liveness vs readiness, dependency probes, status pages, aggregation, flapping control, and load balancer integration.
Health checks tell orchestrators and humans whether an instance should receive traffic. Done wrong, you flap pods out of the pool during blips or keep serving while wedged. Tie this to service discovery, load balancing, and metrics.
Scope with the framework: process liveness, readiness including deps, public status page, and aggregation rules.
Functional requirements
- GET /healthz (liveness) and GET /readyz (readiness).
- Optional deep checks: DB ping, queue depth, disk free.
- Central aggregator builds service-level status from instance reports.
- Public status page with component list and incident history.
- Admin override: force unhealthy for drain.
Non-functional requirements
- Probe endpoints cheap and bounded (hard timeouts).
- Avoid thundering herds against shared DBs every second from every replica.
- Stable signals: hysteresis so one failure does not yank a node.
- Status page available even when the product is down (host separately).
Liveness is not readiness
Liveness = “restart me if I am deadlocked.” Readiness = “do not send traffic yet / anymore.” Checking the DB inside liveness can restart every pod during a DB blip - a classic outage amplifier.
Probe design
| Probe | Checks | On fail |
|---|---|---|
| Liveness | Process up, event loop alive | Restart container |
| Readiness | Warmup done, critical deps OK | Remove from LB |
| Startup | Slow init finished | Hold probes until ready |
Deep dependency checks belong mostly on readiness, with caching (“DB OK for last 5s”) so 100 pods do not stampede. Prefer passive signals (error rate from the mesh) alongside active pings.
Aggregation and status page
- Agents push heartbeat + check results to a central store.
- Rules: component red if >X% instances fail or a synthetic journey fails.
- Status page reads from a separately hosted store/CDN snapshot.
- Incidents: update manually or auto-open when burn rates spike.
Flapping control
- Require N consecutive failures before marking down.
- Require M successes before marking up.
- Jitter probe intervals; align with LB health check settings.
- Circuit-break deep checks if the dependency status service itself is sick.
Worked example
- API pods expose /healthz (noop) and /readyz (cached Redis ping).
- Redis blip: readiness fails after 3 probes; LB drains pods.
- Liveness stays green - no mass restart.
- Status page marks “API degraded” from aggregator rules; clears when ready ratio recovers.
Interview summary
Separate liveness and readiness. Bound deep checks, add hysteresis, host the status page independently. That is health check system design.