Every decision here buys something and charges for it, and the charge usually lands where you were not looking. A cache buys read latency and hands you a staleness problem; a second region buys survival and hands you a conflicting-write problem. Nothing here is simply correct, which is why lists of patterns teach little. What follows is the shape of the trades: what changes when a call crosses a network, which numbers settle a design before any diagram, and why most scaling machinery answers a question your traffic has not asked.
Each chapter opens with the short version. Tap one to read the detail.
What changes when the call crosses a network
~2 min
The mistake that breaks systems is treating a remote call as a slower function call. It has a third outcome a local call cannot: silence. A timeout never tells you whether the work happened, so anything you retry has to be safe to run twice.
A function call returns or throws. A remote call returns, throws, or goes quiet — and the quiet case is different in kind, because the request may have run, run twice, or never arrived, and the timeout tells you nothing about which.
The trap is that retrying looks free. Suppose a payment call times out after ten seconds and your client retries. If the first attempt actually succeeded and only the response was lost, the customer is now charged twice, and nothing in your logs resembles an error. So the discipline is not "retry carefully" but "make it idempotent first": the client attaches a token, the server records it, and a repeat returns the stored result. An operation that is not idempotent cannot be safely retried, and everything is retried eventually.
The second irreducible fact is distance. A round trip inside one building is around a millisecond, across a continent around 45, across an ocean around 70, and a durable disk write costs 1 to 10. Ten sequential calls to a service an ocean away spend most of a second waiting, and faster code recovers none of it. Latency caps throughput too: a sequential protocol paying that durable write tops out near 100 operations per second.
Rule: make an operation idempotent before you make it remote, and count round trips before you optimize code.
Count the work, not the requests
~2 min
One mistake in two costumes: sizing a system by average request rate, and balancing traffic by request count. Requests are not units of work — two can differ by a factor of a thousand — so both decisions rest on a misleading number.
Capacity decisions made before there is traffic usually optimize the wrong dimension, because the number that settles them has not been measured. Start with arithmetic: rate times seconds per day times bytes per record. Suppose you serve half a million requests per second at two kilobytes each: about 86 terabytes a day. One published exercise found a single machine would need roughly 2,500 disks for that write rate, killing the design before anyone built it.
Size for the peak plus a failure, never the average: if a cluster breaks at 5,000 requests per second and your peak is 19,000, you need about six clusters rather than four, because capacity has to absorb losing a failure domain while at peak.
A bigger machine stays one failure domain with a hard ceiling; more machines add coordination. Reads scale by adding copies, writes only by splitting data. Autoscaling is a cost control, not a burst absorber: machines arrive in minutes, spikes in seconds. Scale on a signal that moves with instance count: for a video transcoder 100% processor use is normal, so the honest signal is job-queue depth. Unhealthy instances still drag a fleet-wide average down, and if shedding triggers before scaling does, a rejecting server looks cheap per request — so the balancer sends it more.
The same counting error runs through load balancing. Round robin feels fair and measurably is not: in production it leaves up to a two-times spread in processor use between the least and most loaded task, because query costs vary by a thousand times and identical machines perform differently. Fewest-requests-in-flight lands near the same spread. What narrows it is backends reporting their own utilization and error rate, with clients weighting on that. And a health check returning success without touching its dependencies keeps a broken server in rotation: a process being alive and a service being able to serve are different questions.
Rule: size against the resource that saturates first, and balance on load a backend reports rather than on requests you counted.
Caching is cheap; invalidation is the bill
~2 min
"Add a cache" reads as free performance. A cache without a defined invalidation story is a correctness bug scheduled for later, because every copy raises the question of when it becomes a lie. Below roughly 80% hit rate it is not earning its costs either.
Caching improves read latency dramatically, and the condition arrives in the same breath: you need a clear strategy for when cached data is updated or invalidated, or the system behaves incorrectly. The named anti-patterns are the everyday ones — caching data that changes frequently, treating a cache as durable storage, and never measuring whether it works.
Where the copy sits is the first trade, and the bug is easy to miss in testing. Suppose each client keeps its own cache: that is the lowest latency possible and it keeps serving when the store behind it is down, but two clients now disagree. One caches a result; a minute later another runs the identical query and sees something different, and nothing errors. A shared remote cache removes the disagreement and adds a hop to every lookup.
Expiry is the invalidation that scales, because it needs no messaging — give each item a time to live balancing freshness against pressure on the store behind it, and treat that window as a product decision. The hit rate then tells you whether it is working: aim for 80% or better, and well below that usually means the cache is too small or the key more specific than the data it names.
Distance is the other cache and it has a floor: where users sit limits the latency you can reach. The offsets are a serving presence closer to them and long-lived pooled connections, since a handshake plus slow start makes the first bytes of a short response cost more than the bytes.
Rule: decide the staleness window before you add the cache — a cache with no invalidation plan has only scheduled the bug for later.
Splitting the data, and choosing where it lives
~2 min
Sharding gets treated as a routing problem, so the key is picked for even distribution. If that key is absent from your commonest query, every read fans out to every shard and waits for the slowest — a write bottleneck traded for a tail-latency one.
You split data when one machine cannot take the write rate, and that limit arrives before the capacity limit: write frequency, not total size, is what makes a single disk-backed machine impossible first.
The split key is the real decision and the usual mistake. A key chosen purely for even distribution spreads load beautifully and turns your commonest query into a fan-out across every shard; a key chosen from that query keeps reads cheap but can concentrate load on one popular value. Neither is free. For example, a pipeline joining two streams shards both sides by the identifier it joins on, so matching records land on the same node — split them differently and every join crosses the network.
The store underneath is the same kind of decision, and its anti-pattern is picking one because the team knows it. Write down what the data requires — volume, persistence, atomicity and isolation needs, read-heavy or write-heavy, latency, throughput — and then choose.
Inside a store, every lever reduces bytes read: partition so a query scans a slice, index the columns queries filter on, use a columnar format for wide analytical scans. Read the query plan first — the slow query is rarely the one you assumed.
Rule: choose the split key from the predicate your commonest query already uses, and treat even distribution as a competing goal rather than the default one.
Copies, and what you are willing to promise about them
~2 min
"Eventually consistent" gets heard as "a small delay". It means a user can save something and then not see it. Decide what that does to your product before choosing it — and during a partition you keep answering or keep being correct, not both.
Hardware fails as a matter of routine, so a component with no redundant peer sets the ceiling on availability. Replication is the first answer, and creates a second problem: copies that can disagree.
The framing everyone quotes is consistency, availability and partition tolerance. Partitions are not optional, so the real question is what your system does during one — refuse writes and stay correct, or keep accepting them and reconcile later. Choosing by default means whoever wrote your failover script chose for you.
The misconception hides in the word "eventually". Suppose a user changes a notification setting and immediately reloads. If reads go to any replica they sometimes see the old value, and to them the product has simply lost their change. The fix is not a stronger database — it is routing that user's read to a copy that already holds their write. And the cost lands in application code rather than in the store: teams on eventually consistent systems spend a substantial share of their time building complex, error-prone machinery to cope, none of which appeared in the benchmark that justified the move.
Quorums are how copies agree: with 2f+1 replicas you tolerate f failures, because any two majorities share a member, so nothing already decided can be lost. What that prevents is split brain: promote a replica on a missed heartbeat and a merely slow network gives you two nodes each convinced they lead, both accepting writes.
Rule: decide staleness per read rather than per database, and write down what a user seeing stale data actually experiences before you accept it.
Agreement, and why timestamps are not order
~2 min
Two habits cause most of the damage: electing a leader with heartbeats, and ordering events by timestamp. Neither works, because a slow network looks exactly like a dead peer, and two machines' clocks disagree by more than the events you are ordering.
A consensus protocol gives one guarantee worth its price: the group decides, and the decision survives crashes, delays and lost messages. That makes it right for the few facts that must have exactly one answer — who leads, what the configuration is — and wrong for bulk traffic.
Majorities do the work: with 2f+1 servers you tolerate f failures, and because any two majorities overlap, a newly elected leader holds everything a previous leader committed. An improvised heartbeat scheme cannot promise that.
Ordering comes from a counter, not a clock. Each leadership period carries a number that only increases, and a message from an old period is rejected outright — which is how a server partitioned away learns it is stale instead of acting on dead authority. Election timeouts are randomized because identical timers produce split votes forever, and a leader may answer a read without appending anything only after confirming with a majority that it still leads.
Now the clock mistake. Suppose two services write the same record and resolve conflicts by timestamp. The service whose clock runs a few milliseconds fast wins every conflict, and the loser's write disappears with nothing logged. The same flaw makes timestamps bad keys for recognizing a repeated request — a documented anti-pattern, because clocks skew and two clients can produce the same value.
Rule: order events with a counter you control, and use a proven protocol rather than heartbeats whenever the answer has to be unique.
How services talk: call it, or queue it
~2 min
Two beliefs to drop. Splitting the code makes services independent — not while both halves write the same tables. And a queue removes load — it defers it; if the consumer is slower than the producer, the queue is simply where the failure now accumulates.
The availability argument for smaller services is differentiation: once a page is assembled from many services you can decide which are required. A product page needs its price and details; photos and reviews can be dropped and a customer can still buy. That only works if the boundary is real, and shared data is what makes it fake — two services writing the same tables are one service with extra latency, and cannot be released, scaled or fail independently.
A boundary is made real by a contract: a machine-readable, strongly typed definition of what goes in and out, plus a versioning strategy. Without a schema nothing validates a payload; without versioning every change forces every consumer to release in lockstep. Adding optional fields extends a contract safely; a real version change belongs behind a facade.
Across a boundary you can call or queue. A queue lets producer and consumer meet at a durable layer, so either side can break without taking the other down. The mistake is thinking it removes work; it defers work. Imagine a consumer draining 800 messages per second while producers add 1,000: nothing errors, latency simply climbs until someone notices the backlog. A message older than a few seconds is usually stale because the client has gone, so alarm on the age of the oldest message rather than the depth, and move a permanently failing message aside. Because delivery is at-least-once, the consumer must track the producer's token and pass it downstream.
Rule: a boundary earns its network hop only if the two sides can ship on different days — and a queue is a buffer, never a fix for a consumer that is too slow.
Retries are how a blip becomes an outage
~2 min
Retrying a failed call without backoff and jitter converts a brief blip into a self-inflicted outage: retries are load, and load makes the dependency slower. The follow-on mistake is trying to recover by restarting servers, when the only cure is taking load away.
Work the arithmetic once. Suppose a frontend sends 10,100 requests per second to a backend that serves 10,000. The 100 failures retry a second later, so it becomes 10,200 with 200 failures, then 10,300, and the stream grows until the backend melts. Then the part people miss: even when traffic falls back to 9,000 the system may stay down, because returning a failure is not free either.
Each fix costs something. Randomized exponential backoff, always — without it a single network blip synchronizes every client's retry into one pulse. And a retry budget, which is what actually bounds the damage: allow only 60 retries per minute in a process, or cap retries at 10% of requests, and no number of clients can amplify past that. Retry at one layer only, since four attempts at each of three layers is 64 requests at the database from one user action. Set deadlines rather than timeouts and pass the remaining budget down, or a server works 11 seconds on a request abandoned at 10.
Overload turns metastable when the original trigger is gone but the retries it caused now sustain it. Restarting does not help; recovery means cutting traffic as low as 1% if everything is crash-looping, letting caches warm, then ramping back.
Limits are the preventive half, chosen before the bad day. Limit against the resource that actually runs out, and label work by criticality so every service sheds the same low-value traffic. Rejection must be cheaper than service, or the rejection path becomes the overload. Keep queues short relative to the worker pool: at ten times the thread count with 100-millisecond processing, a full queue silently adds a second of latency.
Rule: bound retries as a fraction of traffic, propagate a deadline, and remember that a system in a retry storm recovers only when you take load away.
Measure what a user feels, not what a server reports
~2 min
Average latency is the reassuring number that hides the problem: a service can average 50 milliseconds while 5% of requests take twenty times that. The second mistake is treating 100% as the goal rather than a budget you are meant to spend.
Three terms get mixed constantly. An indicator is something measured — latency, error rate, throughput, availability. An objective is a target for it over a stated window. An agreement is the external promise with consequences; if missing the number costs nothing, it is an objective.
Averages are the standard mistake. Imagine the typical request takes about 50 milliseconds while 5% take twenty times that: the mean looks healthy, and the unhappy 5% are disproportionately your heaviest users, since more requests per session means more chances to draw a slow one. The median describes the typical experience; the 99th and 99.9th percentiles describe what those users actually get. Often the honest target is layered: 90% under a millisecond, 99% under 10, 99.9% under 100.
Three traps recur. Setting the target at current performance locks in what you happen to do today. Absolutes such as always available cannot be met and get ignored. Too many objectives dilute all of them.
The error budget follows from taking the target seriously: the gap between your objective and 100% is a quantity you may spend on releases and experiments, which turns whether to ship from an argument into a reading. To detect a bad change, prefer metrics a user perceives over resource usage.
Rule: put the window and the measurement method inside the objective, or you have written a wish.
A procedure you have never run does not work
~2 min
A canary compared against yesterday measures the time of day, not your change. And "we have replication, so we have backups" is the expensive misconception — replication copies your accidental deletion to every replica, usually before anyone notices.
A canary is a partial, time-limited deployment plus its evaluation: put 5% of traffic on a change and a 20% error rate becomes 1% overall while you find out.
The part people skip is the control. Comparing after against before makes time the largest variable — release on a Monday, for example, and you are comparing a business day against a weekend. Run a control group on the old version at the same moment. A canary must also outlast your metric aggregation window, so a daily release cannot run a week-long one.
Losing a region is the same discipline pointed at recovery, starting with two numbers that carry separate prices: how long you may be down, and how much data you may lose. Restoring from backups is cheapest, losing hours of data and a day of uptime; a minimal always-on core with data replicated brings that to minutes and tens of minutes; a scaled-down working copy to seconds and minutes; running actively in several places approaches zero on both, and is the most complex because conflicting writes are now your problem.
Which is where the backup confusion costs money. A system that faithfully syncs replicas will faithfully sync a corrupt row or an errant delete to every copy. The layers that help are different in kind: soft deletion keeping removed data recoverable for a window, tiered backups, and continuous validation hunting for corruption. Then rehearse — one published restore ran 5,475 jobs across 1.5 petabytes over seven days, and worked mainly because that team had simulated it.
Rule: rehearse the rollback and the restore on a schedule — they are the two procedures you only run when everything else has already failed.
Written by Keentune. We are not affiliated with or endorsed by the organizations whose documentation informs this guide, and any linked sources belong to their respective owners.
All exam, test, and product names and trademarks are the property of their respective owners and are used here for identification and reference only. Keentune is independent study practice — not affiliated with, authorized, or endorsed by any of these organizations.