I had this professor in school who had a tough spot in his heart for career software engineers. But, being in a hybrid program, he always favoured our classmates in more hardware-focused disciplines like electrical or aerospace engineering.
I like to think he isn't alone in this thinking in his generation which is why we have all these pattern named after mechanical techniques or features. There are few that fit this paradigm much more than circuit breakers.
Before we dig into the concept for the uninitiated, let us explain what software circuit breakers get their inspiration from. Let us not further infuriate our professors with our existence.
An electrical circuit breaker is a device that is used to protect an electrical circuit from damage caused by excess current. It is designed to interrupt the flow of electricity in a circuit if the current becomes too high, which can be caused by a short circuit or an overload.
Which requires manual intervention to restore the system to previous working function.
Circuit breakers, much like their hardware counterparts, are a vital component of their respective systems, helping to improve reliability and resilience by preventing failures from cascading and causing widespread damage. In this post, we'll explore circuit breaker design basics and discuss some best practices for using circuit breakers.
Circuit Breakers
Distributed systems are complex and often include multiple interconnected components, such as servers, databases, and microservices. When one component fails or experiences an issue, it can have a cascading effect on the rest of the system, leading to widespread failures and outages. This is where circuit breakers come in.
Using Circuit Breakers
A circuit breaker is a software design pattern that helps prevent failures from cascading by interrupting traffic flow to a component when it becomes unavailable or starts to experience issues. When the circuit breaker "trips," it interrupts traffic flow to the affected component, redirects it elsewhere, or returns an unavailable error to the requesting client.
How you implement this failure mechanism can be extremely tricky but done correctly can protect core components in your system from catastrophic failure.
Tripping The Circuit
At a high level, circuit breakers work by monitoring the performance of a component and "tripping" if the component starts to experience issues. The specific criteria for tripping the circuit breaker can vary (highly difficult to get write the first time), but it might include things like:
- High error rate.
- Slow response times, or a large number of timeouts.
Engineers often code for the happy path until things go wrong. Thinking through how an upstream service can mess with your system is crucial. Does this request returning timely affect the throughput of your system if it were to slow down? Once you have done that, think through if the circuit breaker for that makes sense or not.
