Why we run per-site cgroups instead of Docker-per-site

When we sketched the multi-tenancy model, the easy answer was 'Docker per site'. A site, a container, an image registry, a deploy. Done. We ran it for three months on a staging cluster and shelved it.

The failure modes were not subtle. Docker layer cache invalidation across hundreds of WordPress sites with slightly different plugin sets means storage cost grows non-linearly. Cold-start latency on a previously-unseen site was 300–900ms — fine for a serverless function, catastrophic for a publisher's homepage. And the kernel work to keep each container's network namespace isolated was, in the end, work we were paying for twice.

The primitive we actually wanted was already in the kernel: cgroups v2, exposed cleanly by systemd 'slice' units. Each site gets a slice that caps CPU, memory, and IO. A Linux user, a PHP-FPM pool, an Nginx server block. The site cannot escape the cgroup. The PHP-FPM pool is named after the user. The user owns nothing but its own document root. We get hard isolation, near-zero overhead, and a deploy step that is `mv` followed by `systemctl reload php-fpm`.

Docker was not wrong; it just was not the right tool for this shape of workload. The lesson, again: Linux has spent thirty years on multi-tenancy. Use it.