Your CRM integration goes down at nine in the morning. There are two thousand contact-sync jobs on the queue, and every one of them is now going to open a connection, wait for a timeout, throw, and get retried. Your workers spend the next hour fully occupied producing nothing, and every other queue behind them is starving.

The API being down is not your problem to fix. Continuing to call it two thousand times is.

Laravel already has a circuit breaker

You do not need to build one. ThrottlesExceptions is job middleware that counts failures and, once there have been too many, stops jobs from running at all for a cooling-off period:

use Illuminate\Queue\Middleware\ThrottlesExceptions;

public function middleware(): array
{
    return [new ThrottlesExceptions(10, 600)];
}

That reads as: if this job throws ten times, stop attempting it for ten minutes.

What it actually does

The behaviour is worth knowing precisely, because it is not quite "the job fails faster".

Before running the job, the middleware asks a rate limiter whether the failure count for this key is already over the limit. If it is, the job is released back onto the queue without handle() being called at all. Nothing touches the dead API.

If the job is allowed to run and succeeds, the failure counter is cleared. One good call closes the circuit again, which is what you want when a service comes back.

If it throws, the counter is incremented and the job is released to try again later.

So the circuit is not a hard stop. Jobs keep cycling through the queue, but while the breaker is open they cost you nothing but a pop and a release.

Tune it for the failure you care about

Out of the box the breaker trips on any exception, which is usually too blunt. A malformed record that throws a validation error should not push you toward opening a circuit that is meant to detect an outage. Use when() to decide what counts:

use Illuminate\Http\Client\ConnectionException;

public function middleware(): array
{
    return [
        (new ThrottlesExceptions(10, 600))
            ->when(fn (Throwable $e) => $e instanceof ConnectionException)
            ->backoff(5),
    ];
}

Anything the callback rejects is rethrown immediately and handled the normal way, so genuine bugs still fail and still reach failed_jobs. backoff(5) sets how many minutes a released job waits before its next attempt.

Share one breaker across related jobs

By default the counter is scoped to the job, which means five different job classes hitting the same CRM each learn about the outage separately. If they share a dependency, give them a shared key:

(new ThrottlesExceptions(10, 600))->by('crm-api')

Now the first job class to notice the outage protects all of them. This is the single most useful option here, and the one most people miss.

You can also report the exceptions that trip the breaker, so an outage still shows up in your error tracker rather than being silently absorbed:

(new ThrottlesExceptions(10, 600))->by('crm-api')->report()

The catch: releases still burn attempts

This is the part that bites. A released job counts as an attempt. If the job has $tries = 3 and the breaker holds the circuit open for ten minutes, the job can exhaust its attempts on releases alone and land in failed_jobs without the API ever having been called again.

The fix is to stop counting attempts and start counting time:

public function retryUntil(): DateTime
{
    return now()->addHours(6);
}

Now the job is allowed to bounce for as long as the outage lasts, and only gives up when the window closes. We go into this trade-off in more detail in job has been attempted too many times or run too long.

Use the Redis variant if you can

ThrottlesExceptionsWithRedis does the same thing with atomic Redis operations. On a busy queue with many workers, the plain cache-driven version can let a few extra jobs through while the counter is being updated. If Redis is already in your stack, prefer it:

use Illuminate\Queue\Middleware\ThrottlesExceptionsWithRedis;

return [(new ThrottlesExceptionsWithRedis(10, 600))->by('crm-api')];

Not the same as rate limiting

A circuit breaker reacts to things going wrong. A rate limiter keeps you inside somebody's published quota, and that is a different problem with a different tool, covered in handling API rate limits in queued jobs.

In practice a busy integration wants both: RateLimited so you stay within the quota when the service is healthy, and ThrottlesExceptions so you back off when it is not.

Wrapping up

When an integration dies, the goal is to stop spending workers on calls that cannot succeed. Add ThrottlesExceptions, scope it with by() so every job hitting that service shares one breaker, narrow it with when() so only connection failures count, and switch the job to retryUntil() so released attempts do not quietly exhaust its tries while it waits.

Related