The data path, shared
Roughly 100 tenant databases per Postgres cluster, 500 per ClickHouse. One node makes that the only option.
The tenant’s code needs its data. CloudNativePG 0.29.0 runs shared Postgres clusters with about a hundred tenant databases in each; ClickHouse runs as a shared StatefulSet with about five hundred databases each.
Why shared. A Postgres cluster per tenant on this cluster means a hundred primaries on one worker node, each with its own shared_buffers, its own WAL, its own pod. It does not fit, and it would not fit on four nodes either. Shared clusters are not a cost optimisation here, they are the only shape that runs.
What that changes about the tenant boundary. Between hops 6 and 7 the boundary changes material. In the pod it is a sandboxed kernel; in the database it is Postgres grants. A database and a role inside a shared server, with REVOKE CONNECT ON DATABASE … FROM PUBLIC, no superuser handed to tenants, and pg_hba written so that a tenant role can reach exactly one database. That is the entire isolation story at this hop — no kernel, no network policy, no separate process. It is worth making a class say that out loud, because a diagram with gVisor on it invites the assumption that isolation is uniform all the way down, and it is not.
Noisy neighbours are structural. max_connections, shared_buffers and the WAL are per-server. One tenant’s connection storm is everyone’s connection storm, and one tenant’s long transaction holds everyone’s vacuum horizon. Pooling is not a performance nicety at this density, it is what keeps a hundred tenants inside one max_connections.
The consequence people meet at the worst moment. Backups and PITR are per-cluster, because the WAL archive is per-cluster. So a point-in-time recovery restores a hundred tenants to the same instant. Restoring one tenant — which is what anyone ever actually asks for — means recovering a whole copy of the cluster to a scratch namespace, then dumping one database out of it and loading it back. That is a procedure with a runbook and an hour of wall-clock, not a kubectl command, and the time to write it down is before someone needs it.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: tenants-01 # ~100 tenant databases in here
spec:
instances: 1 # one node; a second is not schedulable
postgresql:
parameters:
max_connections: "400" # shared across every tenant
backup:
barmanObjectStore: # one WAL archive for all 100 tenants
destinationPath: s3://backups/tenants-01
-- what isolation actually is at this hop:
CREATE DATABASE t_acme;
CREATE ROLE t_acme LOGIN PASSWORD :'pw';
REVOKE CONNECT ON DATABASE t_acme FROM PUBLIC;
GRANT CONNECT ON DATABASE t_acme TO t_acme;
-- no superuser, no CREATEDB, no CREATEROLE, and one database reachable.Check yourself
2 questions. One attempt each is recorded; the explanation is the point, not the score.
A single tenant asks to be restored to 10:00 this morning. What does the shared CNPG cluster imply?
What enforces the tenant boundary at the database hop?