Multi-tenant
Multi-tenant Laravel, part two: per-tenant queues
How Horizon handles per-tenant fairness when one tenant tries to monopolise the workers.
Part ofmulti-tenant-laravelLoading essay…
How Horizon handles per-tenant fairness when one tenant tries to monopolise the workers.
Part ofmulti-tenant-laravelLoading essay…
One tenant uploads a 200,000-row import. The other ninety-nine open the app and wait for their daily sync to finish. With a single global queue, all ninety-nine wait. With per-tenant queues, the noisy tenant only slows itself.
This is part two of the Multi-tenant Laravel series, on what happens to queues when you have customers, not just users.
Out of the box, Horizon gives you a queue per name (default, emails,
sync) and a worker pool per queue. Inside a queue, jobs are FIFO. If
tenant A enqueues 50,000 sync jobs at 09:00 and tenant B enqueues 200
at 09:01, tenant B's jobs sit behind tenant A's. The workers grind
through tenant A first.
This is fair if you think of jobs as the unit of work. It is unfair if you think of tenants as the unit of work, which you should, because your customers do.
We want a guarantee like:
Every active tenant makes at least N jobs of progress per minute, regardless of any other tenant's queue depth.
Round-robin across active tenants gets you most of the way there, with two adjustments:
Concretely, we shard the underlying queue by tenant id and route a custom dispatcher across them:
namespace App\Queue;
use Illuminate\Contracts\Queue\Job;
use Illuminate\Queue\Worker;
class TenantFairWorker extends Worker {
private array $rotation = [];
protected function getNextJob($connection, $queue): ?Job {
$tenantQueues = $this->activeTenantQueues();
if (empty($tenantQueues)) {
return null;
}
// Round-robin: pop the head, look at it, push it to the tail.
$tenantQueue = array_shift($tenantQueues);
$tenantQueues[] = $tenantQueue;
$this->rotation = $tenantQueues;
return $connection->pop($tenantQueue);
}
private function activeTenantQueues(): array {
// Only enumerate queues that have at least one job. Cached for 250ms.
return Cache::remember('tenant-queues:active', 0.25, function () {
return Redis::keys('queues:tenant-*')
->filter(fn ($k) => Redis::llen($k) > 0)
->values()
->toArray();
});
}
}A few decisions worth pulling out:
We name the queues tenant-{id} (e.g. tenant-42). They all live on
the same Redis instance under the same connection. The tenant
boundary is purely the queue name. This means we still get one set of
metrics, one set of dashboards, one place to look.
Earlier versions tried to dispatch jobs across queues in proportion to "who needs work most." That logic ended up duplicating the worker's job. The cleaner cut: producers always push to their tenant's queue, the worker decides who gets served next.
Enumerating Redis keys on every job pop is fine for ten queues, painful for two thousand. A 250ms cache on the active set absorbs the load without making the round-robin feel uneven. (At a job rate of 100/s, you serve 25 jobs from the same snapshot before refreshing. That is fine.)
Round-robin alone has a subtle pathology: if one tenant's job is fast (microseconds) and another is slow (seconds), the slow one starves. The cap fixes this. After serving K jobs from the same tenant in a single rotation, the worker forces a rotation regardless of whether the queue is empty:
protected int $maxConsecutiveJobsPerTenant = 5;
private string $lastTenantQueue = '';
private int $consecutive = 0;
protected function shouldForceRotate(string $tenantQueue): bool {
if ($tenantQueue === $this->lastTenantQueue) {
$this->consecutive++;
return $this->consecutive >= $this->maxConsecutiveJobsPerTenant;
}
$this->lastTenantQueue = $tenantQueue;
$this->consecutive = 1;
return false;
}K = 5 was the sweet spot for us. K = 1 thrashed because Redis pop is not free, and K = 50 starved small tenants when a big one was in mid-job.
Without intervention, a tenant whose jobs all fail and retry will dominate the active set: every cycle, their queue has work, so the round-robin keeps including them, and the failed job retries forever.
The fix: a per-tenant quarantine mechanism. If more than X jobs from a tenant fail in Y minutes, their queue moves to a "slow lane" that gets sampled at 1/10th the rate. Customers reporting issues are suddenly the ones being served, while quiet failure-loop tenants stop hogging.
A 200,000-row import that runs as one job blocks the worker for the whole import. Even with cap-per-cycle, "one job" means "the cap does not trigger." The fix is at the producer side: large imports must be chunked into many smaller jobs at dispatch time. We enforce this with a wrapper:
class TenantImport implements ShouldQueue {
public int $maxRowsPerJob = 500;
public function handle() {
if ($this->rows->count() > $this->maxRowsPerJob) {
// Self-chunk into smaller copies.
foreach ($this->rows->chunk($this->maxRowsPerJob) as $chunk) {
static::dispatch($chunk)->onQueue("tenant-{$this->tenantId}");
}
return;
}
$this->processChunk($this->rows);
}
}Horizon's metrics aggregate jobs across all queues. With per-tenant queues, that aggregate hides the distribution. We added a tenant-by- tenant view: jobs in flight, failure rate, p99 wait time. Without it, "the queue is healthy" can be true on average and false for the specific customer who calls.
A "noisy neighbor" tenant no longer affects everyone else. The escalation that used to come in as "the whole app is slow" now comes in as "we are slow today, every other tenant is fine" — which is a much faster diagnosis.
The cost was about a week of engineering plus the operational discipline to chunk dispatched jobs. Worth it on day one.
The next part is on per-tenant feature flags, the structure that scales past forty flags and the structure that does not.
Series page tracks the rest.