Design a Secrets Manager
How to design a secrets manager for interviews: envelope encryption, rotation, least-privilege access, audit logs, dynamic DB credentials, and break-glass flows.
Secrets managers store API keys, DB passwords, and certificates so they are not baked into images or config blobs. The interview mixes security and distributed systems: encryption, access control, rotation, and high availability for something every boot path depends on.
Clarify with the framework: static secrets vs dynamic short-lived credentials, human vs machine identity, and regional residency.
Functional requirements
- PutSecret / GetSecret / DeleteSecret with versioning.
- IAM policies: who (role/user) can read which path.
- Automatic rotation hooks for DB users and API tokens.
- Audit every access; optional break-glass with extra approval.
- SDK caching with TTL and refresh.
Non-functional requirements
- Encrypt at rest and in transit; keys in an HSM / KMS.
- High availability - secret fetch in the critical path of deploys.
- Low latency for cached GETs; cold KMS unwrap is rarer.
- Tamper-evident audit store.
Envelope encryption
Data keys encrypt the secret blob; a master key in KMS encrypts data keys. Rotation of the master rewraps data keys without rewriting every secret. Say this early - it is the expected crypto sketch.
Architecture
- API fronted by auth / IAM.
- Metadata DB: secret path, versions, ACL pointers, rotation state.
- Encrypted payload store (same DB or object storage).
- KMS/HSM for master keys; workers for rotation lease.
- Clients identify as cloud roles or SPIFFE/mesh identities.
Access path
- Client requests secret://prod/checkout/db with role ARN.
- Authorize against policy; deny by default.
- Load ciphertext + wrapped data key; unwrap via KMS.
- Decrypt payload in memory; return over TLS; scrub buffers.
- Write audit event (who, what, when, from where).
Rotation
- Static: generate new value, dual-publish versions, app flips, revoke old.
- Dynamic DB: plugin creates a short-lived user; leases expire automatically.
- Use a distributed lock so only one rotator runs.
- Clients should re-resolve on auth failure to pick up new versions.
| Secret type | Rotation style |
|---|---|
| Third-party API key | Scheduled overwrite + dual-run window |
| Postgres password | Alternating users A/B |
| TLS cert | Issue new, reload, retire |
| Ephemeral cloud role | STS tokens - no long secret |
Failure and abuse
- Rate-limit GetSecret; alert on fan-out exfiltration patterns.
- Break-glass role time-boxed and heavily audited.
- Replica lag: prefer CP for ACL changes (CAP).
- Never log secret values - redact in logging.
Worked example
- Checkout pods fetch db creds at boot; cache 15 minutes.
- Nightly rotation creates user_v2, updates secret version 14.
- Pods refresh on next TTL or on auth error; user_v1 dropped after grace.
Interview summary
Envelope encryption, IAM on paths, versioned get with SDK cache, and rotation with leases. End on audit and break-glass. That is secrets manager.