Kubernetes Autoscaling for AI Workloads: Stop Paying for Idle GPU

GPU nodes sit at 10–15% utilization while the team debates rightsizing. That's not a tuning problem — it's what happens when you bolt standard Kubernetes autoscaling onto a workload it was never designed for. LLM inference has a fundamentally different resource shape, and the gap between what HPA assumes and what GPU inference actually does is where most AI infrastructure budgets quietly drain away.

This article is about closing that gap with a concrete architecture. Not hand-waving about "right-sizing" — actual decision rules, tool choices, and the tradeoffs teams consistently get wrong.

Why Standard Autoscaling Breaks on AI Inference

HPA scales on CPU and memory by default. LLM inference pegs GPU VRAM and sits at 10–15% CPU. So your HPA sees "low load" while your A100 is saturated and latency is spiking. You either scale too late, or you pre-provision aggressively and pay for the headroom around the clock.

VPA has the inverse problem: it's great at right-sizing request/limit values over time, but it restarts pods to apply changes. A 15-second pod restart during an inference burst is a user-visible outage on a product people are actively using.

Cluster Autoscaler adds nodes reactively — it waits for a pending pod before provisioning. GPU nodes on most cloud providers take 3–8 minutes to become ready. If your burst window is 90 seconds of heavy load, you've already degraded before new capacity arrives.

The result: teams end up holding large "safety" node pools warm at all times. That's not a configuration problem you can tune your way out of. It's an architectural mismatch.

The Three Metrics That Actually Drive AI Scaling Decisions

Before picking tools, get clear on what you're scaling on. The wrong signal — and most teams default to the wrong one — means your autoscaler fires too late or thrashes on noise.

Signal	Best For	Latency Sensitivity
GPU utilization (DCGM)	Batch inference, training jobs	Low — can lag 30–60s
Queue depth (custom metric)	Async agent pipelines	Medium — scale before queue grows
Active request count	Real-time / streaming inference	High — must be near-instant
Token throughput (requests × avg tokens)	Cost forecasting	Not a scaling trigger

For real-time inference — chat, voice, copilots — queue depth or active request count exposed via Prometheus is the right scaling signal. GPU utilization lags actual saturation by tens of seconds, which means you're already in trouble by the time the metric fires.

For async pipelines — document processing, agent batch jobs, embedding runs — queue depth on your job broker (Redis Streams, Kafka, SQS) is usually the cleanest signal. You can scale workers proportionally to backlog before latency degrades, and the scaling decision becomes independent of what's happening inside the GPU.

The trap most teams fall into: they start with GPU utilization because it feels like the most direct signal, then wonder why their scaler always responds too late. Utilization is a lagging indicator. Build your scaling architecture around leading indicators.

The Architecture That Actually Works

Here's the stack I'd wire up for an AI inference deployment that needs to handle burst without burning money on idle capacity.

Layer 1: KEDA for event-driven scaling

KEDA (Kubernetes Event-Driven Autoscaling) replaces HPA for workloads where the right signal lives outside CPU/memory. It supports 60+ scalers — Prometheus metrics, Redis queue length, SQS, Kafka lag. Crucially, it can scale to zero, which HPA cannot.

yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: inference-worker
spec:
  scaleTargetRef:
    name: inference-deployment
  minReplicaCount: 0
  maxReplicaCount: 20
  cooldownPeriod: 120
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring.svc:9090
      metricName: active_inference_requests
      threshold: '4'         # scale up when >4 active requests per pod
      query: sum(active_inference_requests)

The cooldownPeriod is load-pattern-specific. 120 seconds is a reasonable starting point for bursty copilot traffic. For batch processing jobs, push it to 300+ to avoid thrashing on short queue spikes.

Layer 2: Node provisioning strategy

Cluster Autoscaler's reactive model is the wrong fit for GPU workloads. Karpenter provisions node-level capacity based on pending pods and supports consolidation policies that actively remove underutilized nodes — Cluster Autoscaler doesn't do this by default.

The pattern that works: run two node pools. A warm pool of one or two on-demand GPU nodes handles baseline load and the first wave of any burst. A burst pool using spot instances handles spikes — accept that spot can be reclaimed and design your inference workers accordingly, either with checkpointing or job-broker retry logic.

Mixing two or three compatible instance types in your spot pool (for example, g5.xlarge and g5.2xlarge on AWS) dramatically improves the chance the provisioner finds available capacity. Single-instance-type spot pools get reclaimed in correlated waves during regional capacity crunches.

Layer 3: GPU partitioning

This is where most teams leave the most money on the table. GPU nodes are expensive, but many inference workloads don't need a dedicated GPU per pod. NVIDIA MIG (Multi-Instance GPU) partitions a single A100 into multiple isolated slices with hardware-level memory and compute separation. Time-slicing is simpler to configure but gives weaker isolation — if one pod runs a long generation sequence, others feel it.

For smaller models (7B–13B parameters quantized to INT8), a single A100 80GB can run multiple concurrent inference pods with MIG partitioning rather than requiring a dedicated node per workload. For user-facing latency-sensitive inference, prefer MIG's hard isolation. For internal batch pipelines, time-slicing is usually acceptable.

The decision rule: if your workload has strict P95 latency SLOs and multiple tenants or processes sharing the same node, use MIG. If it's internal, async, and latency-tolerant, time-slicing gives you the density benefit at lower configuration overhead.

The Scale-to-Zero Trap on GPU

KEDA's scale-to-zero capability sounds immediately appealing. The problem: a pod that needs to pull a 10–20GB model into VRAM takes 45–90 seconds to go from scheduled to serving-ready, depending on your image caching and model loading implementation. That's not a Kubernetes problem — it's physics. VRAM loading is bounded by PCIe bandwidth.

If you scale to zero on GPU pods without accounting for this, users or upstream agents waiting on that cold start will hit a request timeout before the pod is ready. Fix this with one of two approaches:

1. Don't scale to zero on GPU — scale down to minReplicaCount: 1 during expected quiet hours, full zero only during known maintenance windows.
2. Model caching with persistent storage — use a persistent volume or node-local storage to cache model weights so VRAM loading time drops to seconds instead of re-pulling from object storage on every cold start.

If your agent pipeline has retry logic, a scale-to-zero cold start can trigger cascading retries across your agent graph. The retry storm compounds the latency problem. Wire in an explicit readiness probe on inference pods and make your orchestration layer respect it before routing any traffic.

The Tradeoff Nobody Talks About: Latency vs. Cost Optimization Are Opposing Forces

Here's what most Kubernetes cost optimization content glosses over: every technique that reduces cost on GPU infrastructure introduces latency risk, and teams that optimize for cost first will always be surprised by the latency consequences.

Spot instances save money and introduce reclamation risk. Scale-to-zero saves money and introduces cold-start latency. GPU time-slicing increases density and introduces contention jitter. MIG partitioning isolates tenants and reduces the blast radius of a single heavy workload — but you still have to size the partition correctly upfront, and resizing requires node drain.

The practical framework: sort your inference workloads into two buckets — latency-sensitive (user-facing, synchronous, SLO-bound) and latency-tolerant (async, internal, batch). Apply aggressive cost optimization only to the second bucket. Protect the first bucket with on-demand capacity, MIG isolation, and a minimum replica floor. The teams that try to apply the same cost posture across both buckets end up with degraded user-facing products and an angry on-call rotation.

Cost Decision Table: Which Tool for Which Workload

Workload	Scaling Tool	Node Strategy	Scale to Zero?
Real-time chat / copilot	KEDA (Prometheus)	Warm on-demand + burst spot	No — min 1
Async agent batch jobs	KEDA (queue depth)	Spot-first	Yes — with cold start budget
Scheduled fine-tune / eval	CronJob + node provisioner	Pure spot, terminate after	N/A — ephemeral
RAG embedding pipeline	KEDA (queue)	CPU nodes	Yes
Multi-tenant inference (SaaS)	KEDA + namespace quotas	On-demand with MIG	No

The embedding pipeline row is worth flagging specifically: a lot of teams reflexively put embedding workloads on GPU nodes because they're categorized as AI workloads. For most embedding models — sentence-transformers, self-hosted BGE, or API-based options like text-embedding-3-small — CPU inference is fast enough for batch embedding and costs a fraction of GPU node time. Save the GPU budget for generation workloads where it actually matters.

The Observability Prerequisite

None of this scaling architecture gives you useful signal without the right metrics in place. At minimum:

• DCGM Exporter → GPU utilization, memory used/free, SM activity per pod
• Custom Prometheus metrics in your inference server → active requests, queue wait time, tokens/second, P95 time-to-first-token
• Node provisioner disruption events in your alerting → spot reclamations, consolidations, failed provisioning

Set up a daily cost report that breaks down GPU node-hours per namespace, per model served, per request type. Not weekly — daily. GPU spend can compound inside 48 hours if a misconfigured job runs hot. Weekly reporting means you're always reacting to last week's problem.

If you're running a multi-tenant setup, namespace-level resource quotas need active auditing. You won't see which team is holding idle GPU reservations unless you instrument at that granularity.

What to Actually Do

1. Audit your current GPU utilization with DCGM Exporter this week. If average utilization is below 50% across your inference nodes, you have direct headroom to reclaim — but identify whether it's genuinely idle or masking burst headroom before you cut capacity.
2. Classify your workloads into latency-sensitive and latency-tolerant buckets before touching any autoscaling config. This decision shapes every downstream tradeoff.
3. Replace HPA with KEDA on any inference deployment using a queue, broker, or request-count metric. This is a config change, not an architectural rewrite — you can ship it in a day.
4. Pilot GPU partitioning on one batch or internal inference workload. If you're running 7B–13B models, the experiment takes an afternoon. Validate density and latency impact before rolling it to user-facing workloads.
5. Set a hard scale-to-zero policy for all latency-tolerant workloads: async pipelines, embedding jobs, eval runs. Every GPU-hour these run idle is direct waste with no user-experience justification.

The engineering work here is days. The slow part is getting accurate visibility into your current utilization baseline — that requires instrumentation patience, not months of architecture work. Start with the metrics, then let the data drive the tradeoffs.