There is a specific moment in most AI platform migrations where the infrastructure team realises that everything they know about Kubernetes applies and also doesn’t.
The pods are pods. The services are services. The ingress is ingress. But the GPU node costs $3.50/hour and sits idle 60% of the time because the scheduler doesn’t know that the batch job finishing in seven minutes will free up exactly the capacity the real-time endpoint needs. The model artefact is 14GB and takes four minutes to pull, which means your cold-start time is four minutes, which means your autoscaler is always behind.
GPU scheduling is a different problem
CPU workloads scale horizontally. You need more capacity, you add more pods. GPU workloads do not work this way. A single inference request needs a whole GPU (or a fraction of one, if you set up MPS or MIG, which most teams have not). Scaling from 1 to 2 GPUs doubles your cost. Scaling from 1 to 2 CPU pods increases cost by roughly nothing on a node that has capacity.
The economic implication: GPU utilisation is the single most important infrastructure metric for AI workloads, and most teams are running at 30-40% utilisation because they provisioned for peak and their traffic is bursty.
The artefact problem
Your web app container is 200MB. It pulls in 3 seconds. Your model container is 15GB. It pulls in 4 minutes. Your autoscaler’s scale-up time is now dominated by image pull time, not pod scheduling time. The solutions — pre-pulling images to GPU nodes, using model caching layers, separating the model weights from the application container — are all non-obvious if you come from a web services background.
Spot instances: different strategy for training vs. inference
Training workloads are fault-tolerant (you can checkpoint and resume). Spot instances save 60-70%. Inference workloads serve real-time traffic. Spot interruptions cause user-visible errors. The strategy is different for each, and most teams apply the same strategy to both because their platform doesn’t distinguish between them.
These are platform engineering problems with known solutions, but the solutions are not in the standard Kubernetes playbook. We’ve built AI-native platforms for our own products (Stayligent and AI Doctor) and for enterprise clients across multiple cloud providers. The patterns are documented in our Cloud Native & AI practice.