Why Your LLM Calls Need a Circuit Breaker: A Lesson from Last Party 🔌

Carnival time has begun. 🎉

We are organizing a house party, and if you are in and around Bangalore, you are invited. Just one condition:

Don't be late.

We all have that one friend who is always late. We still invite them. Then we start checking the phone.

"Any minute now."

Meanwhile, everyone is standing around, plans are on hold, food is getting cold, ice is melting, and patience is slowly disappearing. At some point, the problem isn't the late friend anymore.

The problem is that everyone keeps waiting for the late friend.

And surprisingly, distributed systems have the exact same problem. That includes your Agentic Applications. 🤖

A service becomes slow or starts failing, and instead of moving on, the rest of the system keeps waiting for it.

• Requests stay open.
• Threads stay occupied.
• Connections are consumed.
• Queues start growing.

Eventually, a problem in one service starts affecting perfectly healthy services. This is when the Circuit Breaker Pattern comes in.

A Circuit Breaker is literally what it sounds like. If you don't know what one does, let an Electrical Engineer explain it to you. 😎

---

Circuit Breaker in Electrical Engineering ⚡

After barely surviving my B.Tech in Electrical Engineering, I was genuinely terrified the first time I saw Circuit Breaker written as a design pattern.

My brain immediately jumped to power systems, transformers, current, voltage, and those end-semester questions that looked like they were personally designed to fail me. 💀

But here's the funny part: software circuit breakers borrow the same idea we learn in electrical engineering. There are three important states:

Closed, Open, and Half-Open

• A closed circuit means current can flow normally. The switch is ON and power reaches the bulb.
• An open circuit means the circuit is broken. Current cannot flow, so the bulb is OFF.

"Half-Open" isn't really a standard electrical state. For our software analogy, think of it as a testing state. After a fault, instead of allowing everything back immediately, we cautiously allow some flow to check whether the system has recovered. If things are fine, we close the circuit. If the fault is still there, we open it again.

With that analogy in mind, let's move into distributed systems.

---

Circuit Breaker in Distributed Systems 🧱

It's carnival night, so naturally we need pizza. 🍕

Let's say we have a PartyService which needs to call a PizzaService to place the order. Normally, everything looks beautiful:

Host
 │
 ▼
PartyService
 │
 ▼
PizzaService
 │
 ▼
Pizza on the way ✅

Enter fullscreen mode Exit fullscreen mode

The host places an order. PizzaService accepts it. PartyService gets the confirmation. Everyone eats happily.

But what happens when PizzaService starts having a bad day?

• Maybe the kitchen is overloaded (it is carnival night).
• Maybe the service is crashing.
• Maybe the network is slow.
• Maybe someone decided Friday evening was the perfect time to deploy... 👀

Now the flow looks like this:

PartyService
 │
 ▼
PizzaService
 │
 ▼
Request Timeout ❌

Enter fullscreen mode Exit fullscreen mode

One order waiting for a few seconds isn't necessarily a problem. But imagine thousands of hungry guests ordering at once:

1000 Orders
 │
 ▼
Slow PizzaService
 │
 ▼
Threads Waiting
 │
 ▼
Connections Waiting
 │
 ▼
Queues Growing
 │
 ▼
Resources Consumed

Enter fullscreen mode Exit fullscreen mode

And now the interesting part begins. PizzaService is unhealthy. But PartyService can become unhealthy too.

---

The Real Problem: Cascading Failure 🌊

Imagine the following architecture:

                Host
                 │
            PartyService
              ╱       ╲
     PizzaService     DrinkService

Enter fullscreen mode Exit fullscreen mode

Now PizzaService goes down. PartyService keeps calling it. Orders keep waiting. More guests arrive. More threads get occupied. And suddenly PartyService starts running out of resources too.

PizzaService ❌
 │
 ▼
PartyService ❌
 │
 ▼
Whole Application ❌

Enter fullscreen mode Exit fullscreen mode

One service failed, but the failure travelled through the system. This is what we call a cascading failure — and it's exactly what a Circuit Breaker tries to prevent.

---

How Does a Circuit Breaker Work? 🚦

The Circuit Breaker sits between your service and the remote dependency:

PartyService → Circuit Breaker (watches every call) → PizzaService

Enter fullscreen mode Exit fullscreen mode

• If everything looks healthy, requests pass normally.
• If the dependency starts failing repeatedly, the Circuit Breaker says: "Okay, stop calling the service for a while."

And that gives us our three states.

1. Closed State 🟢

The normal state. Requests pass through, and the breaker keeps monitoring.

Request 1 → ✅
Request 2 → ✅
Request 3 → ❌ (network fail, packet drop, kitchens have bad days)
Request 4 → ✅

Enter fullscreen mode Exit fullscreen mode

A few failures don't mean the service is dead. So the circuit stays Closed while the failure rate stays acceptable.

2. Open State 🔴

Now failures start piling up:

Request 1 → ❌
Request 2 → ❌
Request 3 → ❌
Request 4 → ❌

Enter fullscreen mode Exit fullscreen mode

At some point the breaker decides: "Yeah... something is wrong here." 😅

CLOSED --(too many failures)--> OPEN

Enter fullscreen mode Exit fullscreen mode

Now it stops sending requests to PizzaService entirely.

Request → Circuit Breaker → ❌ → Fail Fast / Fallback

Enter fullscreen mode Exit fullscreen mode

The request fails immediately. No unnecessary network call, no waiting for a timeout, no extra load on an already unhealthy service.

3. Half-Open State 🟡

But we can't keep the circuit Open forever. What if PizzaService recovered? After waiting a bit, the breaker moves to Half-Open and lets a few test requests through.

Test Request 1 → ✅
Test Request 2 → ✅ → back to CLOSED 🟢

Test Request 1 → ❌ → back to OPEN 🔴

Enter fullscreen mode Exit fullscreen mode

This is why Half-Open matters. We don't blindly trust the service again — we test it first.

---

The Complete Flow

        CLOSED 🟢
           │
           │  too many failures
           ▼
         OPEN 🔴
           │
           │  wait timeout elapses
           ▼
      HALF-OPEN 🟡
       │       │
  success    failure
       │       │
       ▼       ▼
   CLOSED 🟢  OPEN 🔴

Enter fullscreen mode Exit fullscreen mode

The mental model is simple:

Closed → Let requests through
Open → Stop calling the dependency
Half-Open → Carefully test whether it has recovered

Now let's write one.

---

Let's Build a Circuit Breaker in Java

Using a library without understanding what's happening underneath is a little like driving a car without knowing what the brake does. It works... until it doesn't. 😅

First, our state:

public enum CircuitState {
    CLOSED,
    OPEN,
    HALF_OPEN
}

Enter fullscreen mode Exit fullscreen mode

Now the breaker itself. We track the current state, the failure count, a threshold, how long to stay Open, and when it opened.

import java.util.concurrent.Callable;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicReference;

public class CircuitBreaker {

    private final int failureThreshold;
    private final long openWaitMillis;

    private final AtomicReference<CircuitState> state =
            new AtomicReference<>(CircuitState.CLOSED);
    private final AtomicInteger failureCount = new AtomicInteger(0);
    private volatile long openedAt = 0;

    public CircuitBreaker(int failureThreshold, long openWaitMillis) {
        this.failureThreshold = failureThreshold;
        this.openWaitMillis = openWaitMillis;
    }

    public <T> T execute(Callable<T> call, Callable<T> fallback) throws Exception {
        if (state.get() == CircuitState.OPEN) {
            if (System.currentTimeMillis() - openedAt >= openWaitMillis) {
                state.set(CircuitState.HALF_OPEN);
            } else {
                return fallback.call(); // fail fast
            }
        }

        try {
            T result = call.call();
            onSuccess();
            return result;
        } catch (Exception e) {
            onFailure();
            return fallback.call();
        }
    }

    private void onSuccess() {
        failureCount.set(0);
        state.set(CircuitState.CLOSED);
    }

    private void onFailure() {
        int failures = failureCount.incrementAndGet();
        if (failures >= failureThreshold) {
            state.set(CircuitState.OPEN);
            openedAt = System.currentTimeMillis();
        }
    }
}

Enter fullscreen mode Exit fullscreen mode

Notice what happens when the circuit is Open: we don't call the remote service at all. We simply fail fast.

If we configure new CircuitBreaker(3, 10_000):

3 failures       → OPEN
Wait 10s         → HALF_OPEN
Test succeeds    → CLOSED
Test fails       → back to OPEN

Enter fullscreen mode Exit fullscreen mode

Using it looks like this:

PizzaOrder order = breaker.execute(
    () -> pizzaService.order(guestId),
    () -> PizzaOrder.fallback(guestId)
);

Enter fullscreen mode Exit fullscreen mode

Once the threshold is hit, the next order doesn't even reach PizzaService — it fails fast. The dependency is unhealthy, so we stop feeding it traffic.

---

But This Is NOT Production Ready 👀

Before someone copies that class into a production service... please don't. 😅

It's just enough to understand the concept. Real circuit breakers deal with failure rates, sliding windows, concurrency, timeouts, slow calls, exception filtering, metrics, and fallbacks.

For example, is 50 failures bad?

1000 successful requests + 50 failures  → probably fine
Last 100 calls: 50 failures (50%)       → definitely not fine

Enter fullscreen mode Exit fullscreen mode

A recent window of calls is far more meaningful than a raw count. That's why production apps usually reach for a library.

---

Circuit Breaker vs. Retry vs. Timeout

You might be thinking: "Why not just retry when a request fails?"

Retries solve a different problem.

• A retry says: "Maybe this was temporary. Let's try again."
• A Circuit Breaker says: "We've seen enough failures. Stop calling this dependency."

Imagine PizzaService is already overloaded, you have 10,000 orders, and every order retries three times. Now you're hammering a struggling service with thousands of extra requests. That's basically knocking on your late friend's door four times instead of once. 😂

So distributed systems combine these patterns, each with a different job:

Pattern	Question it answers
Timeout	How long am I willing to wait?
Retry	Should I try again?
Circuit Breaker	Should I stop calling this service?

---

Wait, GenAI Apps Need This Too 💎

Here's the part most people miss.

The War Story That Inspired This Blog ❤️‍🔥

Even I missed this a few weeks back, and it's the real reason this blog exists.

I was building a model orchestrator — a system where one incoming request doesn't just hit a single LLM, it passes through a small pipeline of LLM calls before a final answer comes back:

Request
   │
   ▼
Router LLM   → figures out what the user actually wants
   │
   ▼
Agent LLM    → does the actual task (search, generate, calculate, etc.)
   │
   ▼
Judge LLM    → double-checks the agent's answer before sending it back
   │
   ▼
Final Response

Enter fullscreen mode Exit fullscreen mode

Think of it like a restaurant order going through three people: someone takes your order, someone cooks it, and someone plates it before it reaches your table. If any one of them quietly does nothing and just passes an empty plate along, you still get something — just not what you asked for.

That's basically what happened.

Users started getting weird, inconsistent responses. For the first 30 minutes, I was not worried enough to inspect — the monitoring showed 200 OK for 92% of requests. In most systems, a 200 status means "this worked." So I assumed everything was fine.

It wasn't. When users kept reporting bad answers, we finally looked past the status codes and into the actual response payloads — and found a field I hadn't been watching closely: confidence = 0.

Here's what was actually happening under the hood:

Router LLM   ──fails/times out──▶  falls back silently
Agent LLM    ──fails/times out──▶  falls back silently
Judge LLM    ──fails/times out──▶  falls back silently
                                          │
                                          ▼
                          Response: 200 OK, confidence = 0
                     (looks healthy in the logs — nothing "crashed")

Enter fullscreen mode Exit fullscreen mode

Every single stage of the pipeline was failing and quietly falling back to a default response. No exception. No 500 error. No alert. Just a technically-successful HTTP response carrying a useless answer.

I had built in fallback handling for the case where one LLM call in the chain might fail. What I hadn't considered was the entire chain failing together — because the actual root cause wasn't in my code at all. Around the same time, we started seeing 429 Too Many Requests errors too, and eventually traced it back to a provider-side incident: Anthropic's status page confirmed the model API itself was degraded that day.

The fix was to add a Circuit Breaker at the controller level — the layer that sits in front of all three LLM calls. Before trusting the pipeline with real traffic, the breaker sends a tiny, cheap probe request (just a single token) to confirm the LLM is actually accepting and responding to calls. If the probe fails, the breaker opens immediately and the whole pipeline fails fast with a clear error, instead of quietly returning broken answers dressed up as successes.

That's the real danger with multi-agent systems: a failure doesn't always look like a crash. Sometimes it looks like three fallbacks politely agreeing to lie to you.

---

GenAI is the new fire. Everyone's building on top of an LLM provider — OpenAI, Anthropic, Gemini, a self-hosted model, whatever. And every one of those calls is... a network call to a remote dependency that can be slow, rate-limited, or down.

Sound familiar?

An LLM call is basically the ultimate late friend:

• Responses can take many seconds (streaming tokens one by one).
• Providers return 429 Too Many Requests when you hit rate limits.
• A single agent step can fan out into many model calls (tools, retries, reflection loops) — so one flaky dependency can multiply fast.

Now picture the classic naive setup:

Your AI App → prompt fails or times out → Retry ×3 → Retry ×3 → already rate-limited LLM provider

Enter fullscreen mode Exit fullscreen mode

Retrying a rate-limited model is like ordering more pizza from a kitchen that's already on fire. Every retry pushes you deeper into the rate limit, your latency explodes, and your token bill quietly grows.

A Circuit Breaker in front of the model client fixes this:

       AI Request
           │
           ▼
     Circuit Breaker
     ╱      │      ╲
 CLOSED   OPEN   HALF-OPEN
    │       │        │
    ▼       ▼        ▼
  LLM    Fallback  Test call
Provider            to LLM

Enter fullscreen mode Exit fullscreen mode

And the fallback is where GenAI apps get to be clever:

Model unavailable
├── Return a cached / similar answer
├── Drop to a smaller, cheaper model
├── Return a "please try again shortly" message
└── Queue the request for later processing

Enter fullscreen mode Exit fullscreen mode

So the next time your agent loops and calls the model 40 times in 5 seconds, remember: even the fanciest AI needs a circuit breaker before it keeps knocking on the provider's door.

Same old pattern, brand new fire.

---

What Should a Fallback Do? 🤔

A fallback doesn't mean return null. Please don't.

Dependency Down
├── Return cached data
├── Queue the request
├── Drop to a cheaper path
└── Return a graceful error

Enter fullscreen mode Exit fullscreen mode

The important rule: Don't fake success. If the pizza order failed, don't tell the guest "Order Confirmed ✅" just because your fallback needs to return something.

A graceful failure is much better than a successful lie.

---

One Important Question 🚨

Should every exception open the circuit? No.

A 400 Bad Request (the guest sent garbage input) doesn't mean the service is unhealthy. But these are strong signals that it is:

Connection refused
Timeout
503 Service Unavailable
429 Too Many Requests

Enter fullscreen mode Exit fullscreen mode

So when configuring a breaker, one of the most important questions is:

Which failures should actually count?

That decision is application-specific.

---

Bringing Our Late Friend Back 😊

• Closed 🟢 — Friend: "I'm 5 minutes away." Everyone: "Okay, 👍". The party continues.
• Open 🔴 — After "5 minutes" → "almost there" → "traffic" → "parking" for two hours, you finally say: "Nobody waits anymore." The party moves on without them.
• Half-Open 🟡 — Later: "Bro, are you here?" → "Yes" = party continues. "5 more minutes" = door closed again.

That's the Circuit Breaker.

---

Final Takeaway

Whenever you hear Circuit Breaker, remember the three states:

CLOSED    → requests flow normally
OPEN      → requests stop immediately
HALF-OPEN → a few requests test recovery

Enter fullscreen mode Exit fullscreen mode

Or, in Bangalore house-party terms:

Friend is coming → Everyone waits → Friend keeps failing → STOP WAITING → Party Continues 😎

Enter fullscreen mode Exit fullscreen mode

A Circuit Breaker doesn't fix the failing service. It protects the rest of your system while that service is failing — whether that's a pizza shop, a payment API, or your favourite LLM.

A failure is inevitable. A cascading failure doesn't have to be.

Outro 💚

Firstly, thanks a lot if you're still reading this far. 🎉

Remember, distributed systems are a bit like that one late friend — the failures are inevitable, but how you handle them is entirely up to you. A circuit breaker won't fix a broken service, but it'll stop one bad dependency from taking your whole system (or your whole party) down with it.

The purpose of this blog was to demystify a pattern that sounds intimidating but is genuinely simple once you see it as three states and a timer. Whether it's a pizza order, a payment API, or an LLM call — the same rule applies: stop waiting, fail fast, and test recovery before trusting again.

The more you work with distributed systems, the more you realise the most useful ideas are simple:

• Stop waiting
• Fail fast
• Test recovery before trusting again

And if the answer is no... open the circuit.

I plan to write more on distributed systems patterns and backend engineering — the kind of stuff that shows up in real production systems, not just textbooks.

Till then, go build something that doesn't wait around for late friends. 😄

If you run an organisation and want me to write or create video tutorials, please do connect with me 🤝