Design a Configuration Service
How to design a config service for interviews: versioned keyspaces, environments, push vs poll, validation, rollbacks, caching at the edge, and audit trails.
A config service centralizes runtime knobs - timeouts, pool sizes, endpoint URLs - so you change behaviour without a full release. It sits beside feature flags (flags are typed product switches; config is broader ops data) and service discovery (who to call vs how to call).
Use the framework: multi-env namespaces, CRUD with validation, fan-out to millions of clients, rollback, and audit.
Functional requirements
- Hierarchical keys: app / env / region / key.
- Get(key), Watch(key or prefix), bulk snapshot.
- Publish with schema validation and canary percentage.
- Rollback to a prior version; soft delete with history.
- ACLs: who can edit prod vs staging.
Non-functional requirements
- Read path extremely available - apps should survive control-plane blips on last-known config.
- Propagation in seconds for urgent changes (kill timeouts).
- Strong audit for prod edits.
- Low fan-out cost: do not make every pod hit the DB on every refresh.
Stale config beats no config
SDKs must keep a local snapshot on disk or memory. If the service is down, keep running with the last good version and alert. Same soft-fail idea as discovery.
Architecture
- Control plane API + UI writes versioned documents to a primary store.
- Publisher pushes immutable snapshots to object storage / CDN and notifies listeners.
- Client SDKs poll or stream; apply updates atomically when version increases.
- Optional per-datacenter relays to absorb thundering herds.
| Store | Role |
|---|---|
| Config DB | Source of truth + history |
| Snapshot store | Versioned JSON/YAML blobs (S3) |
| Notify channel | Pub/sub or long-poll “version bumped” |
| Client cache | In-memory + on-disk last good |
Data model
Store (namespace, key) → {value, version, updated_by, checksum, schema_id}. Overrides cascade: global < env < region < host. Resolve by merging layers on read or pre-materializing resolved snapshots per audience.
Safety rails
- JSON Schema / type checks before publish.
- Canary: 1% of pods get vN+1; auto-rollback on error spike (metrics).
- Rate-limit destructive edits; require dual approval for prod.
- Never put secrets here - point to a secrets manager reference instead.
Worked example
- Ops raises http.client.timeout_ms from 2000 to 5000 in prod.
- Publish creates version 88; relays push “namespace checkout@88”.
- Pods fetch snapshot, swap atomically; p99 timeouts drop.
- Error budget burn → one-click rollback to 87.
Interview summary
Versioned snapshots, push/poll fan-out, last-known-good clients, and validation/canary/rollback. Draw a hard line vs secrets and vs feature flags. That covers config service.