Design a Distributed Logging System

Every microservice wants to printf into the void. A logging platform turns that void into searchable history. Interviewers like this prompt because it is a write-heavy ingest pipeline with cheap retention tiers — cousin to metrics monitoring, but with fatter, less structured payloads.

Scope with the framework: agents on hosts, central collection, index + cold storage, search UI/API. Skip building a full SIEM product; mention security log immutability as a stretch.

Functional requirements

• Collect logs from thousands of services/hosts.
• Parse common formats (JSON, syslog); attach service, host, trace_id.
• Near-real-time search and filter (last 15 minutes hot).
• Retention: hot days, warm weeks, cold months.
• Optional: live tail, anomaly alerts, PII redaction.

Non-functional

• Ingest durability: accepted logs should not vanish on collector restart.
• Search freshness: seconds to low minutes for indexed fields.
• Backpressure: when indexers lag, buffer — do not crash app servers.
• Multi-tenant isolation so one noisy customer cannot starve others.

Capacity sketch

Order-of-magnitude: 10K hosts averaging tens of KB/s is already multi‑GB/min; a chatty host at ~500 KB/s makes the math ugly fast (10K × 500 KB/s ≈ 5 GB/s before compression). With ~5–10× compression you still land on terabytes/day. Indexing every field forever is unaffordable — index a curated set (service, level, trace_id) and keep raw blobs for selective scan.

High-level architecture

1. Agent (Fluent Bit / OpenTelemetry) — tail files or receive via SDK; batch + compress.
2. Load-balanced collectors — validate, enrich, apply rate limits per tenant.
3. Kafka (or Pulsar) — durable buffer; partition by tenant/service.
4. Indexer workers — parse, write hot index (OpenSearch/Elasticsearch).
5. Object storage — compressed raw segments for cold retention.
6. Query API — fan out to hot index; fall back to cold scan jobs for old ranges.
7. UI / alerts — dashboards and threshold notifications.

Ingest path

1. App writes structured JSON logs (prefer this over free-text soup).
2. Agent batches (e.g. 1–5 MB or 1–2 s), sends HTTPS to collectors.
3. Collector acks after Kafka produce with required acks.
4. Indexer commits offsets only after durable index/blob write (at-least-once → dedupe by event_id if needed).

At-least-once is the honest default. Exactly-once across agent→Kafka→ES is painful; use idempotent event ids for critical audit logs and accept rare duplicates in debug logs.

Storage tiers

Query path

Parse a Lucene-like query: service:checkout AND level:ERROR AND trace_id:X. Time range prunes indices (daily index pattern). Cap result size; paginate. Expensive queries go to an offline cluster or require sampling — protect the hot cluster like you protect an origin behind a CDN.

Scaling and multi-tenancy

• Kafka partitions and consumer groups scale indexer throughput.
• Per-tenant quotas on ingest bytes/day and indexed fields.
• Shard Elasticsearch by time + tenant hash (sharding).
• Drop or sample DEBUG under pressure; never silently drop ERROR without a metric.

Worked example

1. Checkout service emits {"level":"ERROR","trace_id":"abc","msg":"payment timeout"}.
2. Agent batches; collector writes to Kafka topic logs.checkout.
3. Indexer writes to index logs-2026-07-16 and archives the raw batch to S3.
4. On-call searches service:checkout level:ERROR; finds the line within seconds; jumps to trace system via trace_id.

Interview narrative

Draw agent → collector → Kafka → indexer → hot/cold stores. Emphasize backpressure and retention economics. Contrast with metrics (numbers) and claim the win condition is “debuggable production,” not “index everything forever.”