Bigger server, or more servers?
A busy café has two options when the queue gets long. It can hire a faster barista (but there’s a limit to how fast one person can work), or it can open more counters with more baristas. The second option has no real limit, but only if every barista can serve every customer: if your order is only known to the barista you spoke to, you’re stuck in their queue.
Servers are the same. Scaling up (vertical scaling) means a bigger machine: more CPU, more memory. Scaling out (horizontal scalingHorizontal scalingHandling more load by adding more machines that do the same job, instead of buying a bigger one.Open in glossary →) means more machines doing the same job. Scaling out needs servers that don’t remember anything between requests: stateless servers.
Vertical scaling is simple and often underrated: a single large instance handles a lot, with no distributed-systems complexity, until it hits the largest instance size, a single point of failure, or a cost cliff. Horizontal scaling adds capacity and redundancy linearly, but only for stateless tiers; state moves to purpose-built stores (databases, caches, object storage) that have their own, harder scaling story. In interviews, the expected move is: keep the app tier stateless and horizontally scaled behind a load balancer, then spend the discussion on the stateful tiers.
| Scale up (vertical) | Scale out (horizontal) | |
|---|---|---|
| How | Bigger machine | More machines |
| Limit | The biggest machine you can buy | Practically none for stateless servers |
| If it fails | Everything is down | The others keep serving |
| Code changes | None | Servers must be stateless |
| Cost | Grows fast at the top end | Grows roughly linearly |
Stateless servers
Tap the line with the bug.
@RestController
class CartController {
private val carts = ConcurrentHashMap<String, MutableList<Item>>()
@PostMapping("/cart/items")
fun add(@RequestHeader("X-User") user: String, @RequestBody item: Item) {
carts.getOrPut(user) { mutableListOf() }.add(item)
}
@GetMapping("/cart")
fun get(@RequestHeader("X-User") user: String) = carts[user].orEmpty()
}@RestController
class CartController {
private final Map<String, List<Item>> carts = new ConcurrentHashMap<>();
@PostMapping("/cart/items")
void add(@RequestHeader("X-User") String user, @RequestBody Item item) {
carts.computeIfAbsent(user, u -> new CopyOnWriteArrayList<>()).add(item);
}
@GetMapping("/cart")
List<Item> get(@RequestHeader("X-User") String user) { return carts.getOrDefault(user, List.of()); }
}const carts = new Map<string, Item[]>();
app.post("/cart/items", (req, res) => {
const user = req.header("X-User")!;
carts.set(user, [...(carts.get(user) ?? []), req.body]);
res.sendStatus(204);
});
app.get("/cart", (req, res) => {
res.json(carts.get(req.header("X-User")!) ?? []);
});Hint: It worked perfectly on one server. Now there are three behind a load balancer. Where does the next request from the same user go?
The carts live in this server’s memory. With three servers, the request that adds an item lands on server A, and the request that shows the cart lands on server B, which has never heard of it: the cart looks empty two times out of three. A deploy or crash also wipes every cart on that server.
Move the state to a store every server shares, and the server becomes stateless:
@PostMapping("/cart/items")
fun add(@RequestHeader("X-User") user: String, @RequestBody item: Item) {
redis.opsForList().rightPush("cart:$user", json.encode(item)) // shared by all servers
redis.expire("cart:$user", Duration.ofDays(7))
}@PostMapping("/cart/items")
void add(@RequestHeader("X-User") String user, @RequestBody Item item) {
redis.opsForList().rightPush("cart:" + user, json.encode(item)); // shared by all servers
redis.expire("cart:" + user, Duration.ofDays(7));
}app.post("/cart/items", async (req, res) => {
const key = `cart:${req.header("X-User")}`;
await redis.rPush(key, JSON.stringify(req.body)); // shared by all servers
await redis.expire(key, 7 * 24 * 3600);
res.sendStatus(204);
});What usually has to move out of a server’s memory or disk before you can add a second one:
- Sessions and carts → a shared cache like Redis, or a database (or a signed token the client sends with each request).
- Uploaded files → object storage (S3, Google Cloud Storage), not the server’s disk.
- Scheduled jobs → run once, not once per server: a job scheduler, or a lock so only one server runs each job.
- In-memory caches → fine as a speed-up, as long as the data can be rebuilt and slightly stale copies are acceptable.
Practice: how many servers?
- Peak load: 5,000 requests per second at peak.
- One server’s capacity: load tests show one server handles about 800 requests per second before response times get bad.
- Divide: 5,000 ÷ 800 ≈ 6.3, so at least 7 servers.
- Headroom: plan for one server failing and for traffic growing: 7 + 1 spare, rounded up to 8–10.
- Check the rest: 10 servers each holding 20 database connections is 200 connections. Can the database take that? (The last part of this lesson is about exactly this.)
Where state can live
| State | Options | Trade-off |
|---|---|---|
| Session/auth | Signed tokens (JWT) carried by the client; server-side sessions in Redis | Tokens scale freely but are hard to revoke before expiry; server sessions are revocable but add a lookup per request |
| Shopping cart, drafts | Redis with TTL, or the database | Redis is fast but needs persistence/replication if losing carts matters |
| Files | Object storage + CDN, with pre-signed upload URLs | Clients upload directly, servers never touch the bytes |
| WebSocket connections | Inherently per-server | Route by user (consistent hashing) and fan out via pub/sub between servers |
| Local caches | In-process, per server | N copies, each a little stale; fine for config and reference data |
The usual exception to “stateless” is long-lived connections (WebSockets, gRPC streams): the connection is state. Systems keep the stateful connection layer thin and route messages to it through a pub/sub layer.
- Vertical first is often fine (simple, no distribution) until size, availability or cost limits; horizontal for the stateless tier.
- Stateless app servers: sessions in tokens or Redis, files in object storage, scheduled jobs leader-elected or in a scheduler, local caches only for rebuildable data.
- Long-lived connections (WebSockets) are inherent state: route by user, fan out over pub/sub.
Check this part
What is horizontal scaling?
A server keeps shopping carts in its own memory. What happens with three servers behind a load balancer?
What's the main trade-off between JWT sessions and server-side sessions in Redis?