Design Object Storage (Amazon S3)
How to design S3-style object storage for interviews: buckets and keys, multipart upload, erasure coding vs replication, metadata indexes, consistency, and CDN egress.
Object storage is the blob layer under almost every product: photos on Instagram, video segments for Netflix, backups, CDN origins, and log archives. Interviewers ask it because it forces you to separate a tiny amount of metadata from petabytes of immutable bytes - and to talk honestly about durability, cost, and consistency.
It is not the same problem as Dropbox-style file sync. Here clients speak HTTP PUT/GET to a key, not a filesystem with rename semantics. Scope with the framework: PUT/GET/DELETE, multipart for large objects, strong or eventual read-after-write, Eleven 9s durability marketing.
Functional requirements
- PutObject(bucket, key, body, headers) and GetObject / DeleteObject.
- ListObjects with prefix and pagination (lexicographic keys).
- Multipart upload: initiate, upload parts, complete (or abort).
- Optional: versioning, lifecycle policies (hot → cold → delete), pre-signed URLs.
- Auth: IAM-style identity, bucket policies, encryption at rest.
Non-functional requirements
- Durability first (lose almost no objects), availability second.
- Throughput scales with number of keys and clients; single-key writes are often limited.
- Metadata lookups must be fast; large GETs stream from disk/SSD/HDD tiers.
- Cost predictable: storage GB-month, PUT/GET request counts, egress.
Metadata is the hard part at scale
Bytes are relatively easy once chunked across machines. Finding where an object lives, listing prefixes, and surviving metadata failures are what break designs. Budget time for the index, not only for placing disks.
High-level architecture
- Front door: HTTP API / load balancers with auth and rate limits.
- Metadata service: maps (bucket, key) → list of chunk IDs, checksums, ACL, size, version.
- Data nodes (storage servers): store chunks on local disks; report health.
- Placement / placement-master: decides which nodes hold replicas or erasure shards.
- Optional: multipart staging, lifecycle workers, repair/scrubbers.
Data model
| Store | What it holds | Notes |
|---|---|---|
| Metadata DB | bucket, key, version, chunk list, etag, ACL | Strong consistency preferred for PUT visibility |
| Chunk store | opaque bytes + checksum | Immutable; rewrite = new object version |
| Part index | upload_id → parts | Used until CompleteMultipart |
Pick something that can take high write QPS on metadata (key-value or NewSQL). Keys are often hashed for data placement but stored sorted for ListObjects - or you maintain a separate prefix index. Mention both path-style and virtual-hosted URLs if asked about API shape.
Putting an object
- Client PUTs; API authenticates and computes content hash.
- For small objects: write N replicas or erasure shards to data nodes, then commit metadata.
- For large objects: client starts multipart, uploads parts in parallel to different nodes, then Complete writes final metadata.
- Return etag / version id. Failures before metadata commit leave orphan chunks for GC.
Order matters: data first, metadata second (or two-phase). If metadata commits before all shards land, readers may 404 or get incomplete data. GC workers reap unreferenced chunks from incomplete uploads after a TTL.
Replication vs erasure coding
| Scheme | Storage overhead | Repair cost | When to use |
|---|---|---|---|
| 3-way replication | ~3x | Cheap (copy one replica) | Hot / small objects, simple ops |
| Erasure coding (e.g. 6+3) | ~1.5x | Heavier (reconstruct) | Cold / large objects, cost sensitive |
Interviewers love this trade-off. Hot tiers often replicate; cold tiers erasure-code. Cross-AZ or cross-region copies buy durability against site loss. Tie durability math loosely to independent failure domains - do not invent fake 11-nines proofs.
Consistency
Modern S3 offers strong read-after-write for new objects in a region. In an interview, say: after a successful PUT, metadata is committed so subsequent GETs see the object. Listings and cross-region replication may lag. Overwrites with versioning keep prior versions addressable. Relate choices to CAP: metadata quorum CP-ish, bulk data availability weighted.
Listing and hot prefixes
ListObjects by prefix is a classic hotspot (logs/2026/08/05/...). Shard metadata by hash(bucket+key) for GET/PUT, and maintain a secondary index for lexicographic list, or require clients to spread keys. Same advice as sharding: avoid monotonically increasing keys for every write.
Reads, CDN, and range GETs
GET resolves metadata → streams chunks (possibly parallel). Support HTTP Range for video seekers. Front popular objects with a CDN; origin remains object storage. Pre-signed URLs offload auth to short-lived signatures so browsers talk to storage directly.
Worked example
- User uploads a 5 GB video: CreateMultipartUpload → 50 × 100 MB parts in parallel.
- Each part lands on distinct failure domains; checksums recorded.
- CompleteMultipart writes metadata mapping key → ordered parts.
- Player issues Range GETs; CDN caches popular byte ranges at the edge.
Interview summary
Separate metadata from bytes. Commit data then metadata. Compare replication vs erasure coding with cost. Call out multipart, orphan GC, and prefix hotspots. That is a complete S3-style interview answer.