Design Pastebin (Text Sharing Service)
How to design Pastebin for interviews: short URL generation, paste storage, expiration, access control, and scaling read-heavy text sharing with object storage and caching.
Pastebin is the cousin of the URL shortener — instead of mapping a short link to an existing URL, you store arbitrary text and hand back a short link to read it. Interviewers use it to test object storage choices, TTL policies, and abuse handling without the redirect latency tricks of a URL shortener. If you can explain Pastebin cleanly, you have already shown you understand key-value storage, caching, and read-heavy traffic.
Start with the interview framework: clarify whether pastes can be private, how long they live, and whether users need accounts. Most interviews assume anonymous paste creation with optional expiration — similar scope to bit.ly but with larger payloads.
Functional requirements
- Create a paste from raw text (up to a size limit, e.g. 1 MB).
- Return a unique short URL like `pastebin.com/abc123`.
- Read paste content by short key (public unless marked private).
- Optional expiration: 10 minutes, 1 hour, 1 day, never.
- Optional syntax highlighting is a nice-to-have — metadata flag, not core architecture.
Non-functional requirements
- High availability on reads — links get shared in Slack and forums; downtime is visible.
- Low latency on GET (< 100 ms p99 for text under 100 KB).
- Durability for non-expired pastes — do not lose user data on single disk failure.
- Abuse resistance: malware paste reports, rate limiting on create.
Clarify with the interviewer
Is content searchable on the site (public gallery), or only accessible with the exact link? Private-unlisted pastes are the default assumption — no global search index needed.
Capacity estimation
Assume 10 million new pastes per day, average paste 10 KB. Daily write volume ≈ 100 GB; yearly ≈ 36 TB before replication. Reads often dominate: if each paste is viewed 5 times on average, read QPS is 5× write QPS. A 10:1 read:write ratio is typical — design for cache-friendly GETs.
High-level architecture
- Client POST /api/pastes with body text + optional TTL.
- API servers validate size, run spam checks, generate unique paste_id.
- Store blob in object storage (S3); store metadata row in SQL.
- Return short URL; CDN optional for popular public pastes.
- GET /{paste_id} → lookup metadata → fetch blob → render or return raw text.
Why not stuff everything in PostgreSQL?
Metadata (paste_id, created_at, expires_at, owner_ip, size_bytes, s3_key) fits relational tables with indexes on paste_id and expires_at. The text body belongs in object storage — same split as file storage metadata vs blobs. See SQL vs NoSQL: OLTP row for lookup, S3 for payload.
Generating paste IDs
Reuse patterns from the URL shortener: base62 encoding of a Snowflake ID, or random 8-character keys with collision retry. Paste IDs must be unguessable if pastes are unlisted — avoid sequential integers alone. `paste_id = base62(random_64_bits)` gives vast keyspace.
Expiration and cleanup
Store `expires_at` in the metadata table. On GET, return 404 if `now > expires_at`. Background job (cron or Kafka consumer) scans `expires_at < now` in batches, deletes S3 objects, removes DB rows. Lazy deletion on read is not enough — you must reclaim storage. Index on `expires_at` for efficient sweeps.
Caching strategy
- Redis cache-aside: key `paste:{id}` → full text for small pastes under 64 KB.
- Cache only on read; invalidate on delete or expiry job.
- TTL on cache keys aligned with paste expiration to avoid stale hits.
- Viral paste: CDN caches GET response at edge — same as static content.
API sketch
| Method | Endpoint | Purpose |
|---|---|---|
| POST | /v1/pastes | Create paste; body `{ content, ttl_seconds?, private? }` |
| GET | /v1/pastes/{id} | Fetch raw content; 404 if expired or missing |
| DELETE | /v1/pastes/{id} | Owner-only if auth exists; else report abuse |
Follow REST conventions: 201 on create with Location header, 429 on spam via gateway rate limit.
Scaling and sharding
API tier is stateless behind an L7 load balancer. Shard metadata by hash(paste_id) if single Postgres saturates — pastes are immutable after create, so no hot-row updates. S3 scales horizontally by design. Read replicas on metadata DB for analytics queries (top pastes by views) without touching primary.
Security and abuse
- Rate limit creates per IP and per API key.
- Scan content for malware signatures asynchronously.
- CAPTCHA on anonymous create if abuse spikes.
- Private pastes: long random paste_id is security through obscurity; add password hash in metadata for stronger protection.
Worked example: creating a paste
- User pastes 2 KB of log output, selects 24-hour expiration.
- API generates `paste_id = k7x9Qm2p`, writes metadata row, uploads body to `s3://pastes/k7/k7x9Qm2p.txt`.
- Response: `201 Created`, `Location: /k7x9Qm2p`.
- First GET: cache miss → S3 fetch → populate Redis with 24h TTL → return text/plain.
- Subsequent GETs: Redis hit, sub-5ms latency.
- After 24h: GET returns 404; nightly job deletes S3 object and DB row.
Pastebin vs URL shortener
| Dimension | URL shortener | Pastebin |
|---|---|---|
| Payload | Short URL string only | Full text blob you host |
| Storage | Tiny mapping row | Object storage + metadata |
| Expiration | Often never | Core feature |
| Read pattern | 302 redirect | Return body content |
| Abuse risk | Phishing links | Malware paste, illegal content |
Both share ID generation and caching patterns from our URL shortener walkthrough. Mentioning both in one interview shows you generalize designs instead of memorizing one diagram.
Monitoring and SLOs
- Track create QPS, read QPS, cache hit ratio, S3 GET latency.
- Alert on expiration job backlog (millions of expired rows not deleted).
- SLO: 99.9% availability on GET; creates can degrade briefly during incidents.
- Log paste_id on abuse reports — disable without deleting audit trail.
Storage cost and lifecycle
S3 Standard for hot pastes; transition to Infrequent Access after 30 days if pastes are rarely read. Expiration job is also a cost control — dead data should not sit in Standard tier forever. For text under 1 KB, some teams store inline in PostgreSQL `BYTEA` to avoid S3 overhead; switch to S3 above 4 KB. Mention both — interviewer may ask breakpoint.
View counter: `INCR paste:{id}:views` in Redis, flush to analytics warehouse nightly. Do not update PostgreSQL on every read — that turns a read-heavy workload into write amplification. Popular pastes benefit from CDN; long-tail pastes stay S3 + Redis cache on first access.
MVP vs production
MVP: single region, PostgreSQL + local disk or MinIO, no CDN. Production: S3, Redis, expiration workers, abuse pipeline, multi-AZ. Say which pieces you ship first and what breaks without the rest — interviewers reward phased thinking.
Common follow-up questions
Interviewers often ask how you would add syntax highlighting. Store `language_hint` in metadata; render service runs highlighter on read (stateless) or precomputes HTML on write (faster reads, stale if grammar changes). Another follow-up: custom domains for pastes — CNAME to your service, TLS via managed certificates, same backend.
Global users: replicate S3 cross-region for durability; serve reads from nearest region with metadata lookup routed by geo-DNS. Writes go to home region to avoid conflict resolution — pastes are immutable, so no merge problem. Mention CAP: choose consistency on create confirmation over multi-region write latency.
What to say in the interview
Walk through create and read paths in under two minutes. Emphasize metadata in SQL, blobs in S3, cache on read, TTL job for cleanup, and rate limits on write. Mention how this differs from a URL shortener (you own the content, larger payloads, expiration matters more). That answer hits storage, caching, and ops — without overbuilding search or analytics.