Design a Metrics Monitoring System (Prometheus-style)
How to design a metrics and monitoring system for interviews: metric ingestion, time-series storage, aggregation, alerting, dashboards, and cardinality explosions.
Every sober production story ends with graphs. A metrics system ingests counters/gauges/histograms from thousands of services, stores them as time series, and fires alerts when error rate spikes. It is the cousin of analytics pipelines but optimized for recent, high-resolution operational data rather than long-term warehouse joins.
Clarify with the framework: pull vs push, retention (15 days vs 2 years), and whether logs/traces are in scope (usually metrics only).
Functional requirements
- Ingest metrics with labels (service, host, path, status).
- Query ranges: rate(http_requests[5m]), p99 latency.
- Dashboards over common queries.
- Alert rules → notification channels.
- Service discovery of scrape targets (optional).
Non-functional
- Keep ingesting during partial outages — monitoring must not die with the app.
- Query recent data in under a second for dashboards.
- Survive cardinality mistakes without melting the cluster (or degrade gracefully).
- Retention tiering: hot TSDB, cold object storage.
Cardinality is the boss fight
A label like user_id on every request creates billions of series. Teach clients to use bounded labels (status_code, route_template). Mention this early — interviewers love it.
Pull vs push
| Model | How | Trade-off |
|---|---|---|
| Pull (Prometheus) | Scraper fetches /metrics HTTP endpoints | Simple targets; harder through NAT |
| Push (StatsD/Agent) | Apps push UDP/HTTP to collectors | Easy from everywhere; need buffering |
| Hybrid | Agent on host pushes; central scrape agents | Common in large orgs |
Either works in an interview. Pick one and stay consistent. Push pairs naturally with Kafka as a buffer; pull pairs with a discovery service listing pod IPs.
Architecture
- Instrumentation libraries in each service (counters, histograms).
- Collectors / agents batch and forward samples.
- Ingest gateway — validate, auth, rate limit abusive sources.
- Time-series DB (TSDB) shards by metric name + label hash.
- Query API — PromQL-like evaluation.
- Alertmanager — dedupe, group, route pages.
- Grafana-style dashboard frontend reading the query API.
Time-series storage
A series is identified by metric name + label set. Samples are (timestamp, value) append-only. Store recent hours in memory/mmap chunks; compact to disk blocks; expire by retention. Downsample old data (1s → 1m → 1h) to save space — similar lifecycle thinking to Pastebin TTL tiers.
- Shard by hash(series_id) across ingest nodes (sharding).
- Replication factor 2–3 for durability of recent data.
- Compression (XOR delta, Gorilla-style) for float time series.
Aggregation and rollups
Dashboards rarely need raw 1-second points for a 30-day chart. Pre-aggregate rollups in the background, or compute rate() at query time over raw samples for short windows. Histograms need careful merge rules (same bucket boundaries) — say “use sparse histograms” if pressed.
Alerting
- Rule: `rate(http_5xx[5m]) / rate(http_requests[5m]) > 0.05` for 2m.
- Evaluator runs rules on a schedule against the TSDB.
- Alertmanager groups by service, silences maintenance windows, pages Slack/PagerDuty.
- Idempotent notification delivery — same ideas as the notification system.
Failure modes
- Scrape target down → mark series stale; alert on up{} == 0.
- Ingest hotspot on one metric → isolate or sample.
- Query of death (huge regex on labels) → timeouts and query budgeting.
- Monitoring of the monitoring: meta-metrics on ingest lag and TSDB disk.
Capacity math
10K services × 100 metrics × 10 label combos = 10M series. One sample every 15s → ~670K samples/sec. At 16 bytes compressed ≈ 10 MB/s ingest — very doable with a small Kafka + TSDB cluster. Cardinality mistakes turn 10M into 10B; that is the real scaling threat.
Logs and traces (boundary)
Metrics answer “how much / how fast.” Logs answer “what happened to request X.” Traces answer “where time went across services.” Keep them separate stores with shared trace_id / request_id correlation. Trying to build one mega-observability DB in an interview usually loses the plot — say three pillars, one design deep.
SLO math
Error budget = 1 − SLO. Alert on burn rate, not raw error count, so a brief blip does not page at 3 AM. Tie alerts to user journeys (checkout success) when possible.
Worked example
- Checkout service exposes http_requests_total{path,code}.
- Agent scrapes every 15s; samples land on TSDB shard 7.
- On-call opens dashboard; query API computes 5m error rate.
- Rule fires; Alertmanager pages the checkout rotation once (grouped).
Interview summary
Draw agents → ingest → TSDB → query/alert. Emphasize labels, cardinality, pull vs push, and retention. Separate monitoring from business analytics warehouses. That distinction keeps the design crisp.