A payment provider's API once started answering in eight seconds instead of eighty milliseconds. It wasn't down — just slow. Within four minutes, a checkout service that had run two years without an incident went fully dark. Request threads piled up waiting on a dependency that never said no; it just never said yes fast enough.
That's the failure mode most teams never plan for: the slow dependency, not the dead one. We design for the happy path, add a load balancer, and call it resilient. But resilience isn't something you buy. It's a set of decisions about how your system behaves when — not if — a piece of it breaks.
Failure is the default, not the exception
Once you're past a single box, something is always partially broken. A disk is filling, a node is rebooting, a network link is dropping one packet in a thousand. Do the math: a request that touches 30 services, each 99.95% available, has a best-case success rate around 98.5% — meaning roughly 1 in 65 requests hits at least one degraded dependency. At a million requests a day, that's a lot of angry users.
The shift that matters is mental. Stop asking "how do I keep this from failing?" and start asking "when this fails, what happens next?" Everything below flows from that question.
The patterns that earn their keep
You don't need a service mesh and a PhD. Four patterns cover most of the blast radius:
- Timeouts on every network call. The default timeout in most HTTP clients is effectively infinity. That's how one slow upstream exhausts your thread pool. Set aggressive, explicit timeouts — and make them shorter than the caller's timeout, so failures surface at the edge instead of stacking in the middle.
- Retries with backoff and jitter. Naive retries turn a hiccup into an outage: the moment a service recovers, every client hammers it at once. Exponential backoff spaces attempts out; random jitter stops them from synchronizing into a thundering herd. And never retry a non-idempotent write blindly.
- Circuit breakers. When a dependency is clearly down, stop calling it. A breaker trips after N failures, fails fast for a cooldown window, then lets a trickle of traffic test the waters. This protects both you and the service you'd otherwise be DDoSing during its recovery.
- Bulkheads. Isolate resources so one drowning dependency can't take the ship down. Separate thread pools or connection pools per downstream mean a stuck recommendations call can't starve the checkout path of workers.
Degrade on purpose
The goal isn't "never fail." It's "fail in a way the user barely notices." That's graceful degradation, and it's a product decision as much as an engineering one.
Serve slightly stale data from cache when the source of truth is unreachable. Show generic recommendations instead of a blank shelf. Queue a non-critical write and confirm it later. Put feature flags in front of expensive, optional work so you can shed load under pressure by turning things off rather than falling over. In my engagements, the teams that recover fastest are the ones who decided in advance which features are load-bearing and which are garnish.
You don't have resilience until you've broken it yourself
Failover that has never been triggered is not a safety net — it's a theory. I've watched a perfectly documented database replica sit unused for a year, only to fail promotion during the one incident it was built for, because a config had silently drifted.
Untested failover is just a hope with a YAML file attached.
Fix that with practice. Run game days: schedule an hour, kill a node in staging (or production, if you're brave and prepared), and watch what actually happens. Inject latency, not just outages — slow is harder than dead. Load-test past your expected peak so you know where the cliff is before your customers find it for you.
Where to start Monday
Don't boil the ocean. Pick your single most important user flow — the one that costs you money or trust when it's down. Then:
- Map its dependencies and put an explicit timeout on every one.
- Add backoff-with-jitter to retries, and one circuit breaker around your least reliable dependency.
- Decide, on paper, how that flow should degrade — and build the fallback.
- Run a 60-minute game day this month to prove it works.
Resilience isn't a milestone you reach; it's a muscle you keep training. The question worth sitting with this week: if your most critical dependency got slow — not gone, just slow — right now, would your users even notice?