System Design Advanced

04  Reliability & Failure Modes

Everything fails: networks flake, processes die mid-request, disks lie. Reliable systems are not the ones that avoid failure -- they are the ones that fail without making anything worse, and recover without a human awake.

Step 1 / 2

Your checkout service calls a third-party payment provider. Sometimes the call times out -- and you cannot tell whether the charge went through. A naive retry occasionally double-charges customers. Design the safe retry: how the request becomes idempotent, how you retry (how many times, spaced how), and what happens when the provider stays down.

Hints
  • The core trick: the SAME logical charge carries the same identifier on every attempt, so the provider can deduplicate.
  • Retries: bounded attempts, exponential backoff, and jitter. Why does everyone retrying on the same schedule make an outage worse?
  • A timeout is not a decline. When retries are exhausted, what state do you record, and which process later settles the truth?
Your design is reviewed against a hidden rubric.