Circuit breakers: fail fast instead of waiting on a dead service
TL;DR
A circuit breaker sits between a caller and its dependency. When failures cross a threshold it “opens” and fails subsequent requests immediately instead of letting them hit timeouts. After a cool-down it lets a trial request through to check whether the service has recovered.
A dependency starts slowing down or dies. Every request to it waits for the full timeout (say 1 second) before failing. Meanwhile your threads, connections and memory are held up, queues grow, and your service slows down too: a local failure spreads into a cascading outage. A circuit breaker solves this by sitting between the caller and the dependency: when errors pass a threshold, it "trips" and fails subsequent requests immediately instead of letting them wait for the timeout.
Three states
| State | Behaviour | Moves to |
|---|---|---|
| Closed | Requests pass through normally; the breaker counts errors. | Open when the error threshold is reached (e.g. 5 errors) |
| Open | Fails immediately without calling the service, leaving it alone to recover. | Half-open after a wait (e.g. 3 seconds) |
| Half-open | Lets 1 trial request through. | Closed on success, Open again on failure |
In the video: during the 3 seconds the circuit was open, 15 requests got an immediate response instead of each waiting 1000ms.
Why it helps
- Fail fast: users get a response immediately instead of waiting for a timeout.
- Protects your resources: no threads/connections tied up in calls that will surely fail.
- Gives the dependency a chance to recover: less load while it's in trouble.
- Isolates failures: stops them spreading to other parts of the system.
Sketch code
class Breaker {
state = 'closed'; fails = 0; openedAt = 0;
async call(fn) {
if (this.state === 'open') {
if (Date.now() - this.openedAt < 3000) throw new Error('circuit open');
this.state = 'half-open';
}
try { const r = await fn(); this.state = 'closed'; this.fails = 0; return r; }
catch (e) {
if (++this.fails >= 5 || this.state === 'half-open') { this.state = 'open'; this.openedAt = Date.now(); }
throw e;
}
}
}Choosing parameters
- Error threshold: consecutive errors or error rate within a time window; you need enough traffic for a rate to be meaningful.
- Open duration: too short and you keep hitting a service that hasn't recovered; too long and the outage drags on.
- Trial requests in half-open: few, to avoid overwhelming a recovering service.
- Which errors count: only infrastructure errors/timeouts, not business errors like 404.
How to apply it
- Put a breaker on every outbound call (third-party APIs, other internal services) so one dead service doesn't bring down the whole system.
- Have a fallback when the circuit is open: return cached data, a default value or a friendly message.
- Combine with short timeouts and retries with jitter; the breaker stops pointless retries when the service is down.
- Monitor: alert when a breaker opens; it's an early incident signal.
Suggested libraries: Resilience4j (Java), Polly (.NET), opossum (Node.js), or a service mesh such as Istio.
Sources & references
- CircuitBreaker — Martin Fowler