System Design Advanced

04  Reliability & Failure Modes

Everything fails: networks flake, processes die mid-request, disks lie. Reliable systems are not the ones that avoid failure -- they are the ones that fail without making anything worse, and recover without a human awake.

Step 2 / 2

Architecture review: one load balancer, two app servers, one PostgreSQL primary, nightly backups stored in the same data center. List this system's single points of failure and biggest failure modes, give the fix for each, and define three SLIs you would monitor, each with a numeric target.

Hints
  • Walk the request path component by component and ask: what happens if this dies? Do not forget the data center itself, or the backups.
  • The database is the big one. With nightly backups only, how much data do you lose in the worst case?
  • SLIs are measurements: availability percentage, a latency percentile, error rate, replication lag. Each needs a number.
Your design is reviewed against a hidden rubric.