Redis memory keeps climbing. You check the queues and they are empty, or near enough. Nothing is backed up, nothing is stuck, and yet the instance is using several gigabytes more than it did last month.

If you run Horizon, the memory is probably not your queue at all. It is Horizon's record of jobs that already finished.

Horizon keeps a copy of everything

To draw its dashboard, Horizon stores information about jobs as they move through the system: recent jobs, completed jobs, failed jobs, and the tags used to monitor them. That data lives in the same Redis your queue uses, and it outlives the jobs themselves.

On a low-volume app you will never notice. Push a few million jobs a day through it and the history is much larger than the working queue ever was.

The trim settings are in minutes

This is where the surprise usually is. config/horizon.php has a trim block, and the values are minutes, not hours or days:

'trim' => [
    'recent' => 60,
    'pending' => 60,
    'completed' => 60,
    'recent_failed' => 10080,
    'failed' => 10080,
    'monitored' => 10080,
],

So out of the box Horizon keeps an hour of recent and completed job data, and seven days of failures. Read 10080 as 60 * 24 * 7 and the numbers stop looking arbitrary.

An hour sounds modest until you multiply it by your throughput. At a thousand jobs a minute, that is sixty thousand job records retained at any moment, each carrying its payload.

What to change

Lower the high-volume ones, keep the diagnostic ones:

'trim' => [
    'recent' => 20,
    'pending' => 20,
    'completed' => 10,
    'recent_failed' => 10080,
    'failed' => 10080,
    'monitored' => 2880,
],

completed is nearly always the one worth cutting first, because successful jobs are the bulk of your volume and the least interesting to look at afterwards. Failures are rarer and far more useful, so leave them alone.

If you use tags heavily, monitored is worth watching too, since monitored tags retain job data independently of the other settings.

Metrics need the scheduled snapshot

The dashboard's throughput and runtime graphs come from periodic snapshots, and those only exist if the command is scheduled:

$schedule->command('horizon:snapshot')->everyFiveMinutes();

Forget it and the metrics pages stay empty. How many snapshots are retained is separate from job trimming:

'metrics' => [
    'trim_snapshots' => [
        'job' => 24,
        'queue' => 24,
    ],
],

memory_limit is not your workers' memory

A quick clarification, because the name misleads. This setting in horizon.php:

'memory_limit' => 64,

applies to the Horizon master process, the coordinator. It is not the per-worker memory limit. That one lives on each supervisor's config as memory, and controls when a worker recycles itself, which is the setting that matters for climbing worker memory.

The eviction policy that eats jobs

Worth checking while you are in there, because this one is a data-loss bug rather than a housekeeping one.

If your Redis is shared with your cache and configured with an eviction policy like allkeys-lru, Redis is free to delete any key when memory runs short. That includes queued jobs. They do not fail, they do not land in failed_jobs, they simply cease to exist.

A Redis holding a queue should use noeviction, so it refuses writes rather than silently discarding work. Better still, give queues their own Redis database or instance so cache pressure and queue durability are not competing for the same memory. There is more on what Redis persists in persisting Redis data on disk.

Wrapping up

Growing Redis memory with an empty queue is usually Horizon's history rather than your jobs. Trim completed and recent hard, keep failed long enough to be useful, schedule horizon:snapshot if you want the graphs, and make sure the Redis holding your queue is not allowed to evict keys under pressure.

Related