Multi-Tenant K8s: The Namespace Cost Leak Nobody Audits

Multi-tenant Kubernetes clusters look cheap on paper — one control plane, shared node pools, unified observability. In practice, the hidden cost is tenants subsidising each other in ways your billing dashboard will never surface. The pattern I see repeatedly: teams spend months tuning HPA settings while a single noisy tenant quietly burns 30–40% of shared GPU capacity on unthrottled batch jobs. The dashboard shows "cluster utilisation" — which looks healthy — and nobody notices until the invoice arrives.

This article is about the specific cost mechanics of multi-tenant clusters: where money leaks, why standard namespace isolation doesn't stop it, and the concrete controls you need before you add your next tenant.

Why Namespace Isolation Is Not Cost Isolation

Namespaces give you RBAC boundaries and network policy scopes. They do not give you billing boundaries. Three things routinely leak across namespace walls:

Compute overcommit with no per-tenant accounting. Kubernetes schedules on requests, not limits. If Tenant A sets cpu: 100m requests but cpu: 2000m limits, they can burst into headroom that Tenant B's pods are counting on. The scheduler says "fine" — because the node has capacity at request time. The contention shows up as throttling or eviction, not as a line item.

Egress costs are invisible until they're enormous. Cloud providers charge per-GB for egress, cross-AZ traffic, and NAT gateway throughput. In a shared cluster, egress from every tenant flows through the same NAT gateway or cloud LB. Your cloud bill shows one number. You have no idea which tenant's data pipeline to S3 is responsible for 60% of it.

Persistent volume waste compounds silently. PVCs don't get garbage-collected when a namespace is quiet. Tenants spin up staging environments, run tests, then forget the volumes. At $0.10–$0.20/GB/month on gp3 in most clouds, 10 tenants each accumulating 500GB of orphaned volumes is a predictable, recurring drain — that's $500–$1,000/month in pure waste before you've written a single line of product code.

The Right Mental Model: Tenant = Cost Unit

Stop thinking about namespaces as security perimeters and start treating them as billing ledgers. Every control you add should answer one question: can I attribute this dollar to a tenant?

That framing changes what you instrument. You're not just asking "is the cluster healthy?" You're asking "who is spending what, and is that spend proportionate to their allocated share?"

This is the same principle behind showback/chargeback in FinOps — but most Kubernetes implementations stop at cluster-level showback. Per-tenant showback requires labelling discipline from day one, and most teams add it retroactively, which means retrofitting labels onto 200 existing deployments. Don't do that. Set the standard before the second tenant arrives.

Four Concrete Controls That Actually Work

1. LimitRanges + ResourceQuotas as a pair, not individually

ResourceQuotas cap total namespace consumption. LimitRanges enforce per-container defaults. You need both. A ResourceQuota without LimitRanges means a single container can consume the entire quota. A LimitRange without a ResourceQuota means tenants can create unlimited pods up to the per-container limit.

yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: tenant-defaults
  namespace: tenant-a
spec:
  limits:
  - type: Container
    default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "100m"
      memory: "128Mi"
    max:
      cpu: "4"
      memory: "8Gi"

yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "8"
    requests.memory: "16Gi"
    limits.cpu: "32"
    limits.memory: "64Gi"
    persistentvolumeclaims: "20"
    requests.storage: "500Gi"

The persistentvolumeclaims and requests.storage caps are the ones people forget. Include them.

2. Node pools per tenant tier, not per tenant

Full node-per-tenant isolation is expensive and operationally heavy unless you're running a regulated SaaS. The practical middle ground is tier-based node pools: a standard pool for most tenants, a compute-heavy pool for tenants running batch or ML workloads, and a burstable pool backed by spot/preemptible instances for non-critical jobs.

Use nodeSelector or nodeAffinity to route tenants to the right pool. Combine with taints so the compute-heavy pool doesn't get saturated by standard workloads:

yaml
# Node taint
kubectl taint nodes -l pool=compute-heavy dedicated=compute-heavy:NoSchedule

# Pod toleration
tolerations:
- key: "dedicated"
  operator: "Equal"
  value: "compute-heavy"
  effect: "NoSchedule"

This approach means your cost attribution per pool is clean — cloud billing tags the node pool — and you can right-size each pool independently. The gains from putting batch workloads on spot nodes are large; the catch is you need your job scheduler (Argo Workflows, Kueue, or similar) to handle preemption gracefully.

3. Egress metering with eBPF-based tools

This is the most underutilised control in multi-tenant clusters. Tools like Cilium's Hubble or Retina use eBPF to observe network flows at the pod level without injecting sidecars. You get per-namespace, per-pod egress byte counts that you can feed into your FinOps dashboard or alert on.

The alternative — waiting for the cloud bill — means you're always one month behind. With flow-level metering you can catch a tenant whose nightly export job just tripled in data volume before it hits the invoice.

For cross-AZ traffic specifically, label your pods with the availability zone (most managed K8s providers inject topology.kubernetes.io/zone) and use topology-aware routing to prefer same-zone endpoints. In clusters with chatty microservices spread across AZs, cross-AZ data transfer costs are frequently one of the top three line items on the cloud bill — topology-aware routing is the lowest-effort lever to pull.

4. PVC lifecycle automation

Orphaned volumes are a solved problem that most teams just don't solve. The pattern:

• Label every PVC with tenant, environment, and created-by
• Run a weekly CronJob that identifies PVCs not mounted by any pod for >7 days and emits an alert (or auto-deletes in non-prod)
• Set a namespace-level storage quota (shown above) so runaway staging environments hit a wall instead of silently accumulating

For the auto-cleanup CronJob, you need a service account with list and delete permissions on PVCs scoped to the relevant namespaces — not cluster-wide. Least privilege here is both a security and an audit concern.

The Labelling Standard You Can't Retrofit

Every workload in a multi-tenant cluster needs these labels at deployment time:

Label	Example value	Why it matters
tenant	acme-corp	Cost attribution
environment	prod / staging	Quota policy routing
team	platform	Escalation path
cost-center	cc-1042	Finance chargeback
managed-by	helm / argocd	Drift detection

Enforce this with an admission webhook (OPA Gatekeeper or Kyverno). A policy that rejects deployments missing required labels sounds aggressive until the third time you're trying to explain an anomalous cloud bill with no attribution data. Make the label requirement the default; make exceptions explicit and audited.

The resource request/limit gap is one of the most common misconfigurations in multi-tenant clusters — a single tenant with misconfigured requests can displace others from the scheduler without any warning signal in standard monitoring.

The Showback Loop: Turning Data Into Behaviour

Attributing cost is necessary but not sufficient. The reason cost attribution works in engineering teams is that it creates accountability without bureaucracy — if a team sees their namespace cost spike in a weekly Slack digest, they investigate. If nobody sees it, the behaviour doesn't change.

The minimal viable showback loop:

1. Export namespace-level metrics (CPU, memory, storage, egress) to your observability stack — Prometheus/Grafana or your cloud provider's native tooling
2. Map namespace → tenant → cost-center using a config file versioned in git (not a spreadsheet)
3. Generate a weekly cost digest per tenant. Keep it simple: current week vs. prior week, top three cost drivers, any quota approaching 80%
4. Route anomalies (>20% week-over-week spike) to a Slack channel the tenant team owns

Tools like Kubecost or OpenCost automate most of this. OpenCost is CNCF-graduated and free; Kubecost adds more enterprise features. Either is better than trying to build your own cost model from raw Prometheus metrics — that's a trap teams fall into repeatedly, spending weeks on the accounting layer instead of shipping product. The visibility alone tends to change team behaviour; imagine a team that had no per-namespace breakdown suddenly seeing that their staging environment is costing more than their production workload.

Multi-Tenant AI Workloads: A Specific Warning

If your cluster runs LLM inference or GPU workloads alongside standard web services, the stakes are higher. GPU nodes are 5–15x the per-hour cost of CPU nodes, and the noisy-neighbour problem is worse because GPU memory is not overcommittable — a model loaded into VRAM by Tenant A is memory Tenant B cannot use, regardless of their quota.

The correct architecture for multi-tenant GPU workloads:

• GPU nodes in a dedicated pool with NoSchedule taints
• Time-slicing or MIG (Multi-Instance GPU) partitioning where workloads permit (inference servers with small batch sizes benefit; large batch training does not)
• Kueue for queue-based fair scheduling across tenants with configurable quotas per tenant per GPU type
• Hard limits on GPU requests in the namespace ResourceQuota

Time-slicing increases utilisation but introduces latency jitter. For latency-sensitive inference, MIG partitioning on A100/H100 is cleaner — fixed memory, guaranteed isolation. The tradeoff is less flexibility in how you size the partition.

This matters beyond cost: if you're routing requests to self-hosted models and a batch tenant is saturating the GPU, your p99 latency degrades badly and your SLA is at risk. Cost and reliability are the same problem at the infrastructure layer.

What to Actually Do

1. Audit your current PVC inventory this week. Run kubectl get pvc -A and cross-reference against running pods. Find every volume not mounted by an active workload. The number will surprise you.

2. Add the five labels above to every new deployment, enforced by Kyverno policy, before the next tenant onboards. Retrofitting is painful; the policy takes a day to write.

3. Deploy OpenCost or Kubecost in read-only mode first. Get the data visible before you try to act on it. Two weeks of baseline data is enough to identify the top cost drivers.

4. Set up topology-aware routing if you're running across multiple AZs. This is a Kubernetes feature (topologyKeys in EndpointSlices) and a node label — not a rewrite.

5. Create a tier-based node pool structure before you need it. Once you're running batch and real-time workloads in the same pool, separating them requires pod migrations. Design the pools upfront.

Multi-tenancy is an operations multiplier in both directions — it scales your platform efficiently when the controls are right, and it amplifies cost chaos when they're not. The controls aren't complex. The discipline to apply them before you need them is the hard part.