Kubernetes meets AI: hardware is back in the spotlight
Kubernetes made us forget about servers. AI is making us remember them. As generative models and agentic apps roll into production, the bottleneck isn't just code - it's GPUs, interconnects, and the way we schedule and observe all of it.
The consensus at KubeCon + CloudNativeCon: training may stay with a few giants, but inference is everyone's problem. It's showing up in customer apps, internal tools, and CI/CD. It's latency sensitive, cost sensitive, and hungry for the right hardware.
Why inference changes your stack
"We're now in a place where we have to consider 400-gig networking because the models need stuff like that," said Joep Piscaer. The old "throw GPUs at it" mindset doesn't hold up once you're fighting p95 latency and unpredictable spikes.
The new target is hardware-agnostic pipelines: build once, run on GPUs, TPUs, or edge accelerators without rewriting everything. Projects like SynergAI point to this future - Kubernetes orchestrating across mixed hardware and cutting quality-of-service hits by more than 2x.
Vendors and platform teams are rethinking the stack
Cloud platforms such as Google Kubernetes Engine are leaning into accelerator awareness and fleet-level scheduling. Expect tighter integration with device plugins, topology-aware scheduling, NUMA hints, and model snapshotting to reduce cold starts.
"Inference is going to happen on hardware that people are touching, but it's going to be Kubernetes built into all this stuff intuitively," said Savannah Peterson. Translation: plan for fine-grained control across GPUs, fast storage, and the network fabric - not just more compute.
Observability grows up for AI
Rob Strechay shared a question he keeps hearing: "How do I do observability for prompts?" Traditional dashboards stop at CPU, memory, and request counts. That's table stakes now.
Ned Bellavance put it plainly: "We have these golden signals we normally observe for… Now there's a new metric we have to watch, which is the prompt and also the response." Teams are starting to instrument prompts, responses, token usage, and retrieval accuracy with platforms such as OpenTelemetry and eBPF-powered tooling. The goal: fix issues in production fast and tune cost and latency with real data.
Kubernetes adapts to AI's second arc
Kubernetes isn't just about containers anymore. Inference workloads need GPU scheduling, accelerator-aware orchestration, and high-speed networking as first-class concerns.
The CNCF's new Certified Kubernetes AI Conformance Program sets expectations for GPU/TPU scheduling, telemetry, and cluster behavior for AI-heavy workloads. Google Cloud's GKE Pod Snapshots aim to cut inference startup time by up to 80% - a big deal for cold-start pain.
Kelsey Hightower once noted Kubernetes was built with a ~20-year horizon. At 11 years in, the real question isn't "what comes after K8s?" It's "what does K8s look like when AI is the main workload?" The answer taking shape: Kubernetes as the central nervous system for AI - smarter scheduling, predictive scaling, and AI-native observability.
Practical next steps for IT, platform, and dev teams
- Audit the network. Profile end-to-end throughput and tail latency. Where inference is critical, plan upgrades toward 100G-400G and low-latency fabrics.
- Go hardware-agnostic. Abstract accelerators via device plugins and resource classes so the same pipeline can run on GPUs, TPUs, or edge devices.
- Make accelerators schedulable. Use topology-aware scheduling, GPU partitioning, queueing, and quotas to prevent noisy neighbor issues.
- Instrument AI behavior. Track prompts, responses, tokens/sec, time-to-first-token, cache hit rates, and retrieval quality with OpenTelemetry and eBPF.
- Set SLOs for inference. Define p95/p99 latency, cost per 1K tokens, and accuracy targets tied to real user flows.
- Reduce cold starts. Use model snapshotting, warm pools, and fast local storage. In GKE, evaluate Pod Snapshots or equivalents on your platform.
- Control spend. Autoscale on tokens/sec or queue depth, use quantization (e.g., int8/int4) where acceptable, and right-size models for each route.
- Plan for edge. Run smaller models close to users for latency-sensitive paths and fall back to larger models in the cloud as needed.
Key quotes worth bookmarking
- Joep Piscaer: "We have to consider 400-gig networking because the models need stuff like that."
- Rob Strechay: "How do I do observability for prompts?"
- Ned Bellavance: "There's a new metric we have to watch, which is the prompt and also the response."
If you're upskilling teams for AI production work - from prompt telemetry to accelerator-aware Kubernetes - here's a curated set of learning paths by role: AI courses by job.
Your membership also unlocks:
