Per-tenant feature flags without spaghetti
Three years of running flags. The structure that scaled. The structure that did not.
Loading essay…
Three years of running flags. The structure that scaled. The structure that did not.
Loading essay…
A feature flag is a single line of branching logic that someone needed once. By the time you have forty of them across a codebase serving a hundred tenants, "needed once" has compounded into something that nobody fully understands. This essay is about the structure that prevented that compounding for us, and the structure that did not.
Three honest uses, in order of how often they are right:
Anything else, you probably do not need a flag. A/B testing is its own problem with its own tools. Long-running flags that nobody dares remove are the problem this essay is about.
We landed on a single table:
CREATE TABLE feature_flags (
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
key VARCHAR(120) NOT NULL, -- 'bulk_export'
description TEXT NOT NULL, -- the why
owner_team VARCHAR(60) NOT NULL, -- who removes this
rollout_kind ENUM('global', 'plan', 'tenant', 'percent') NOT NULL,
rollout_value JSON NOT NULL, -- per-kind payload
sunset_date DATE NOT NULL, -- the death sentence
created_at TIMESTAMP NOT NULL,
UNIQUE KEY (key)
);Three details that earn their keep:
owner_team is required#A flag without an owner is a flag nobody removes. The first version of
this table left it nullable; we ended up with thirty flags and no
clarity on who could turn any of them off. Making it NOT NULL and
seeding "platform" as a default forced the conversation.
sunset_date is required#Every flag, even permanent ones, has a date by which it must be
revisited. "Revisited" can mean removed, can mean confirmed, can mean
extended. The point is that the table is not write-only. We run a
scheduled job that emails owners 14 days before sunset; if a flag
sunsets without revision, it gets a STALE tag and shows up red on
the platform dashboard.
rollout_kind and rollout_value are normalised#Every flag is one of four shapes:
global — value: {"enabled": true}. Boolean for everyone.plan — value: {"plans": ["pro", "enterprise"]}. Tier gate.tenant — value: {"tenant_ids": [42, 113, 7]}. Explicit list.percent — value: {"percent": 25}. Stable hash by tenant id.By forcing every flag into one of four shapes, we got two things: a single evaluation function instead of forty, and the ability to list "all flags currently enabled for tenant 42" with one query.
namespace App\Features;
class Features {
public function isOn(string $key, ?Tenant $tenant = null): bool {
$flag = $this->cache->remember(
"flag:{$key}",
60,
fn () => FeatureFlag::where('key', $key)->first()
);
if (!$flag) {
return false; // a flag we forgot about is off
}
$value = $flag->rollout_value;
$tenant ??= app(TenantContext::class)->current();
return match ($flag->rollout_kind) {
'global' => (bool) ($value['enabled'] ?? false),
'plan' => $tenant && in_array($tenant->plan, $value['plans'] ?? [], true),
'tenant' => $tenant && in_array($tenant->id, $value['tenant_ids'] ?? [], true),
'percent' => $tenant
&& (crc32("{$key}:{$tenant->id}") % 100) < ((int) ($value['percent'] ?? 0)),
};
}
}Three things to notice:
crc32("{key}:{tenant_id}")
is deterministic per (flag, tenant). The same tenant gets the same
answer every time, even after restarts. This is how you do gradual
rollouts without flapping users in and out.We tried two structures that broke at around twenty flags.
FEATURE_BULK_EXPORT=true, FEATURE_DARK_MODE=enterprise, etc. Worked
for a while. Failed when we needed per-tenant gating: env vars cannot
say "tenant 42 yes, tenant 113 no" without an array, and the array
became unreadable. Also, every change required a deploy.
A config_overrides JSON column on the tenants table. Worked at
five flags. At fifteen, the column was a wall of {"feature_x": true, "feature_y": false, "feature_z": true, ...} for every tenant,
with no central place to ask "what does feature_x mean" or "who
owns it." The structure put the data on the tenant, but flags belong
to features, not tenants.
Every Friday morning, a Slack bot posts the list of flags whose
sunset date is in the next 30 days, with their owners. The owner has
a week to either: confirm and extend, remove the flag (delete the
table row plus the Features::isOn calls), or convert it to a
permanent capability (add to the plan table).
The first month we did this, we removed nine flags. Three of them had been on for over a year with no objections. Half of those had been "experimental" features long since shipped to everyone.
After eighteen months on the new structure, we had forty-six active flags. The dashboard fits on a screen. New flags take five minutes to add (insert + reference). Old flags get retired on schedule. No engineer has needed to ask "what does feature X mean" in over a year.
The ritual is more important than the table schema. The schema is just the surface that makes the ritual possible.
The next part of the multi-tenant series is on tenant data exports without breaking the database. Series page tracks the rest.