Design an API Gateway
How to design an API gateway for interviews: routing, auth, rate limiting, SSL termination, request transformation, and how it differs from a load balancer.
Every diagram has a box labeled "API Gateway" — fewer candidates can explain what it does differently from a load balancer. The gateway is the front door to your microservices: it terminates TLS, authenticates callers, enforces rate limits, routes `/users` to the user service and `/orders` to the order service, and returns consistent error shapes. This article gives you language that sounds like someone who has shipped APIs, not just memorized AWS product names.
Pair this with REST API design and the interview framework. Gateways implement policies; they should not contain business logic like calculating shipping tax.
Gateway vs load balancer
| Concern | Load balancer (L4/L7) | API gateway |
|---|---|---|
| Primary job | Distribute traffic to healthy backends | Route requests by path/header and apply policies |
| TLS | Often terminates SSL | Terminates SSL + may validate JWT |
| Routing | Host/path to pool | Path to service + version + canary weights |
| Auth | Usually none | API keys, OAuth, mTLS |
| Rate limits | Sometimes basic | Per client, per route, per tier |
| Examples | AWS ALB, NGINX | Kong, Apigee, AWS API Gateway |
In practice, managed clouds merge layers: ALB + API Gateway + WAF. In interviews, draw them as two boxes so you can discuss responsibilities clearly.
Core responsibilities
- Request routing — map external URL to internal service cluster.
- Authentication and authorization — validate JWT or API key before backend work.
- Rate limiting and quota enforcement — protect downstream databases.
- Request/response transformation — header injection, JSON ↔ gRPC transcoding.
- Observability — access logs, metrics, distributed tracing IDs.
- SSL/TLS termination — certificates managed at the edge.
Request path (walk through aloud)
- Client sends `GET https://api.example.com/v2/orders/42` with Bearer token.
- Gateway terminates TLS, assigns `X-Request-Id`, checks WAF rules.
- Rate limiter bucket for `client_id` — 429 if empty.
- Auth plugin validates JWT signature and `exp` claim.
- Router matches `/v2/orders/*` → order-service Kubernetes service.
- Optional: strip `/v2` prefix, add `X-User-Id` header from JWT claims.
- Forward to healthy pod via internal load balancer; return response unchanged or normalized.
Routing strategies
- Path-based: `/users` → user-svc, `/payments` → payment-svc (payment design).
- Header-based: `X-Api-Version: 2` → v2 cluster.
- Canary: 95% traffic to stable, 5% to new version — weighted routing.
- Geographic: route EU users to `eu-west` cluster for CAP latency wins.
Authentication patterns
API keys in header for server-to-server and developer portals. OAuth 2.0 bearer tokens for user sessions. Gateway validates JWT locally with public keys (JWKS) — no call to auth service per request if token is self-contained. For invalid tokens, return 401 without hitting backend. Internal service mesh may use mTLS instead; gateway is the trust boundary for public internet.
Rate limiting at the edge
Enforcing limits at the gateway protects all services at once. Store counters in Redis with token bucket Lua scripts — same design as our rate limiter article. Different limits per route: `POST /login` stricter than `GET /public/status`. Return `X-RateLimit-Remaining` headers per REST best practices.
High availability
- Run multiple gateway instances behind an L4 load balancer (yes, LB in front of gateway).
- Config is declarative (YAML/CRD) synced from Git — no manual drift.
- Health checks on gateway itself; circuit break to unhealthy backends.
- Cold start: gateway stateless except Redis rate-limit keys — scale horizontally.
When backends are overloaded
Circuit breaker: after N failures to order-service, gateway fails fast with 503 instead of queueing threads. Bulkhead: separate connection pools per upstream so one slow service does not exhaust all sockets. Mention timeout budgets — client sees 30s timeout but gateway aborts at 5s and returns 504.
Anti-patterns to avoid
Do not put business logic in the gateway
Calculating discounts or joining user + order tables in gateway Lua scripts creates a distributed monolith. Gateway policies only: auth, route, limit, log.
- Chaining 12 sync HTTP calls inside gateway for one client request — use BFF or GraphQL layer instead.
- Storing session shopping cart in gateway memory — use Redis.
- Different error JSON per service without normalization — clients suffer.
Plugin model (Kong / Envoy style)
Gateways extend via plugins: auth, rate limit, logging, CORS, request transformation. Each plugin runs in a defined order (auth before rate limit before route). Custom plugins in Lua (Kong) or WASM (Envoy) for company-specific rules. Keep plugins stateless; state lives in Redis. This architecture lets platform team ship new policies without redeploying every microservice.
| Plugin | Typical order | Purpose |
|---|---|---|
| Request ID | 1 | Generate correlation ID |
| Auth | 2 | Validate JWT / API key |
| Rate limit | 3 | Token bucket in Redis |
| ACL | 4 | Route-level role check |
| Proxy | 5 | Forward to upstream |
| Response transform | 6 | Normalize errors |
Service mesh (Istio/Linkerd) adds sidecar proxies with similar policies inside the cluster — gateway handles north-south (internet to cluster), mesh handles east-west (service to service). Junior candidates conflate them; senior candidates draw both layers.
BFF and GraphQL (when gateway is not enough)
Mobile app home screen needs user profile + orders + notifications in one screen. Three REST calls triple latency. Backend-for-frontend (BFF) service aggregates upstream calls server-side, or GraphQL gateway resolves one query graph. Position BFF behind the gateway: gateway handles auth and limits; BFF handles composition. Do not merge BFF into gateway config — separate deploy cycles.
Observability at the edge
- Structured access logs: method, path, status, latency, client_id, request_id.
- Propagate `X-Request-Id` to all downstream services for distributed traces.
- Metrics per route: p50/p99 latency, 4xx/5xx rates, rate-limit denials.
- WAF integration for SQL injection and bot signatures before traffic hits backends.
Worked example: login abuse
Attacker hammers `POST /v1/login` with credential stuffing. Gateway rule: 10 requests per minute per IP on `/login`, stricter than global 1000/min. Failed auth returns identical 401 body (no user enumeration). After threshold, CAPTCHA challenge plugin runs before forwarding. Backend database never sees 100K bogus attempts — gateway absorbed them. Tie this story to rate limiter design token buckets.
Multi-region gateways: deploy gateway clusters per region with shared Redis for rate limits (CRDT or centralized Redis with cross-region latency trade-off). Route users via geo-DNS. Config propagation via GitOps — same route definitions everywhere, different upstream service discovery per cluster.
Zero trust and mTLS (internal APIs)
Public gateway terminates TLS from browsers. Inside the cluster, gateway-to-service calls may use mutual TLS so a compromised pod cannot impersonate the gateway. SPIFFE/SPIRE issues short-lived certs. Mention this when interviewer asks "how do you prevent lateral movement" — separates mid-level from senior answers.
API keys for partner integrations rotate via secrets manager; gateway hot-reloads keys without restart. Per-partner rate limits and audit logs satisfy enterprise contracts.
Interview summary
Explain gateway as policy enforcement point vs LB as traffic distributor. Walk one request through auth, rate limit, route, forward. Name Redis for limits, JWT for auth, circuit breakers for resilience. Reference how your e-commerce or chat design exposes public APIs through this layer. That is a complete gateway answer in five minutes.