Resilience Patterns

Circuit Breakers, Bulkheads & Graceful Degradation

Resilience Patterns | Technical Operations Excellence

N+2
Redundancy
3
Circuit States
<1s
Timeout Target
5
Defense Layers

Circuit Breaker Pattern

Prevents cascading failures by stopping requests to failing services.

Closed

Normal operation, requests pass through

Open

Requests blocked, return fallback immediately

Half-Open

Limited test requests to check recovery

Bulkhead Pattern

Like watertight compartments in ships - isolate failures to prevent sinking.

TypeMechanismUse Case
Thread PoolDedicated poolIsolate slow deps
SemaphoreConcurrency limitLightweight isolation

Graceful Degradation

Reduce work or quality to maintain availability during failures.

StrategyExample
Quality ReductionLower image resolution
Feature SheddingDisable recommendations
Subset QuerySearch cache only
Default ResponseReturn static content

Retry with Backoff

delay = min(maxBackoff, base * 2^attempt + jitter)

  • Do: Add jitter, cap max delay, limit attempts
  • Don't: Retry non-idempotent ops, nest retries

Load Balancing: L4 vs L7

AspectL4 (Transport)L7 (Application)
RoutingIP + PortHTTP headers, URLs
Latency10-100 µs0.5-3 ms
CPULowHigh (TLS)
Best ForDDoS, non-HTTPSmart routing

Production: Layer both (L4 edge → L7 internal)

Timeout Strategy

TypeTypical Value
Connect250ms - 1s
Header5 - 30s
Idle30 - 300s

Critical: Timeouts DECREASE deeper in call chain

Defense in Depth

Multiple independent layers - no single layer is exclusively relied upon.

  1. Prevention of abnormal operation
  2. Control of abnormal operation
  3. Control within design basis
  4. Control of severe conditions
  5. Mitigation of consequences

Cascading Prevention

PatternPurpose
TimeoutsBound waiting time
BulkheadsIsolate resources
Load SheddingReject before instability
DeadlinesPropagate time limits

Fail Fast, Recover Faster

Every pattern protects downstream dependencies.