A worker has been on the same job for four minutes. The CPU is idle, the database is fine, and nothing is in the log. It is not stuck in a loop, it is sitting on an open socket waiting for a geocoding API that is never going to answer.

While it waits, that worker is unavailable for anything else. A handful of jobs like this and your whole queue looks broken.

Laravel's HTTP client does set a timeout

If you use Laravel's HTTP client, you are not completely exposed. It applies defaults:

  • timeout of 30 seconds for the whole request
  • connect_timeout of 10 seconds just to establish the connection

So a hung endpoint gives up after half a minute. That is a sane default, and it is more than a lot of code gets.

The exposure comes from everything else. A raw Guzzle client you built yourself, a vendor SDK, a cURL call, a file_get_contents against a URL: several of those will wait indefinitely by default. One of those in a job is how you get a worker parked for an hour.

// no timeout of its own, will wait as long as the socket stays open
$body = file_get_contents('https://geo.example.com/lookup?q='.$address);

The number that actually decides it

Here is the interaction people miss. Your worker has its own timeout, and it defaults to 60 seconds.

Do the arithmetic for a job that enriches three addresses in a loop. Each call is allowed 30 seconds. Three slow calls is 90 seconds, and the worker kills the process at 60. The job dies part-way through, having done some of the work, and gets retried.

Nothing here is misconfigured on its own. The per-call timeout is reasonable, the worker timeout is the default, and together they guarantee a job that can never finish on a bad day.

So set the per-call budget from the job's total budget, not the other way round:

public int $timeout = 120;

public function handle(): void
{
    foreach ($this->addresses as $address) {
        $response = Http::connectTimeout(5)
            ->timeout(20)
            ->retry(2, 1000)
            ->get('https://geo.example.com/lookup', ['q' => $address]);
    }
}

Three addresses, up to 20 seconds each with two retries, comfortably inside a 120 second job timeout.

connectTimeout and timeout are not the same

connectTimeout covers getting a connection open. If the host is unreachable or the DNS is dead, this is the one that fires, and it should be short. Five seconds is generous for a connection that is going to work.

timeout covers the whole exchange including the response body. This is the one to size against how slow the endpoint legitimately is.

Splitting them means a dead host fails in five seconds while a genuinely slow report still gets its twenty.

Keep retry_after above the job timeout

One more number has to agree. If retry_after on the connection is shorter than the job's timeout, the queue decides the job is lost and hands it to a second worker while the first is still waiting on that socket. Now two workers are doing the same job.

The ordering to keep is: retry_after longer than the job timeout, and the job timeout longer than the sum of its calls. We cover the first half of that in Laravel queue configuration keys explained and what goes wrong in prevent your queued jobs from running twice.

What a worker timeout actually does

When the worker's alarm fires it kills the process. That is a hard stop: there is no unwinding, no finally block, and no chance to tidy up. Whatever the job had half-written stays half-written.

This is a good reason to keep jobs short and to make them safe to run twice, because a job killed at second 60 will be retried and will repeat whatever it managed to do before it died.

Catching the timeout yourself

If you want to react rather than fail, catch it. Laravel throws ConnectionException for a timeout:

use Illuminate\Http\Client\ConnectionException;

try {
    $response = Http::timeout(20)->get($url);
} catch (ConnectionException $e) {
    $this->release(60);

    return;
}

If timeouts are frequent rather than occasional, releasing every job is just a slower way to hammer a struggling service. That is the point at which a circuit breaker is the better answer.

Wrapping up

A worker that looks hung is nearly always blocked on I/O. Give every outbound call an explicit timeout, keep connectTimeout short and timeout sized to the endpoint, and make sure the job's own timeout is bigger than everything it might wait for while retry_after is bigger than that again. Numbers that do not agree are how one slow API takes your queue with it.

Related