Enterprise AI on Kubernetes: Production Readiness Checklist

This checklist covers the operational, security, and reliability requirements you must address before running enterprise AI workloads on Kubernetes in production. It focuses on the practical gaps between a working demo and a system that can run 24/7 with real users, real data, and real consequences.

Background: Why Kubernetes for AI?

Kubernetes has become the de facto orchestration platform for enterprise AI because it provides a consistent API for deploying, scaling, and managing containerized applications. AI workloads—training jobs, model serving, batch inference—are compute-intensive and often need GPU or specialized hardware. Kubernetes abstracts the underlying infrastructure, allowing teams to run the same workload on-premises, in the cloud, or in hybrid environments. However, the same flexibility introduces complexity: you must handle dynamic resource allocation, multi-tenancy, and the unique lifecycle of AI models (training, versioning, rolling updates, rollback).

Core Requirements: The Non-Negotiables

Before you even think about AI-specific features, your cluster must meet basic production standards. These are the same requirements as any enterprise application, but with AI-specific twists:

• High Availability (HA): Run at least three control-plane nodes and three worker nodes across failure domains. For AI, also consider GPU node pools with dedicated taints and tolerations.
• Backup and Disaster Recovery: Back up etcd, persistent volumes (PVs), and your model registry. Test restores regularly. For AI, model weights and training datasets are often too large for frequent snapshots; use object storage (S3, GCS) for those and back up only metadata.
• Observability: Deploy Prometheus for metrics, Loki for logs, and Jaeger or Tempo for tracing. AI-specific metrics include GPU utilization, memory, and thermal throttling—use the NVIDIA DCGM exporter or equivalent.
• Security: Enable RBAC, network policies, pod security standards, and image signing. For AI, also scan model artifacts for malicious code (e.g., pickle files) and enforce model provenance.
• Resource Governance: Define ResourceQuotas and LimitRanges per namespace. AI workloads often have bursty resource needs; use Vertical Pod Autoscaler (VPA) for training jobs and Horizontal Pod Autoscaler (HPA) for serving.

AI-Specific Infrastructure: GPUs, TPUs, and Scheduling

Kubernetes needs device plugins to expose accelerators (NVIDIA GPUs, AMD ROCm, Google TPUs) as allocatable resources. The standard is the NVIDIA device plugin, which advertises GPU counts and memory. For production, you must consider:

• GPU Sharing and MIG: Use Multi-Instance GPU (MIG) for A100/H100 to partition GPUs for smaller models, or use time-slicing for development. For production, MIG is preferred over time-slicing because it provides better isolation.
• Bin-Packing vs. Spread: For training, you want bin-packing to maximize GPU utilization; for serving, you want spread to avoid a single point of failure. Use node affinity, taints, and tolerations to enforce this.
• Job Scheduling: The default Kubernetes scheduler is not enough for multi-node training jobs (e.g., PyTorch Elastic, TensorFlow distributed). Use a job scheduler like Volcano, Kueue, or KubeFlow Training Operator to handle gang scheduling, queueing, and fair-share.
• Node Pools: Separate GPU nodes from CPU nodes. Use spot/preemptible instances for fault-tolerant batch jobs, but never for stateful serving.

Model Serving: Patterns and Pitfalls

Model serving is the most common production AI workload. The key decisions are:

• Server Type: Choose between a general-purpose server (Triton, TensorFlow Serving) or a serverless framework (KServe, Seldon Core). The latter provides autoscaling, canary rollouts, and multi-model serving.
• Autoscaling: Use the Kubernetes HPA with custom metrics (e.g., requests per second, GPU utilization) or KEDA with event sources like Kafka. For LLMs, also consider scale-to-zero to save costs, but be aware of cold-start latency.
• Model Versioning: Store models in a registry (MLflow, S3) and use immutable tags. In Kubernetes, deploy a new model version as a new ReplicaSet and shift traffic gradually using a service mesh (Istio) or a progressive delivery tool (Argo Rollouts).
• Inference Optimization: Use model quantization, batching, and caching to reduce latency. For LLMs, use continuous batching (vLLM, TensorRT-LLM) to improve throughput.
• GPU Memory: Monitor GPU memory closely; OOMKilled is the most common failure. Set requests/limits for GPU memory (via device plugin) and enable memory oversubscription for CPU-only models.

Data and State: Persistent Volumes and Pipelines

AI workloads are data-hungry. You need a storage strategy that balances performance and cost:

• Storage Classes: Use SSDs for training data and model checkpoints; use HDDs or object storage for archives. For shared datasets, use a distributed filesystem (CephFS, GlusterFS) or a network filesystem (NFS) with proper performance tuning.
• Stateful Workloads: For databases (feature stores, vector DBs), use StatefulSets with stable persistent volumes. For training, use ephemeral volumes for temporary data and PVs for checkpoints.
• Data Versioning: Use tools like DVC or LakeFS to version datasets and models. In Kubernetes, this means mounting the correct dataset version as a volume.
• Pipeline Orchestration: For end-to-end ML pipelines (data preprocessing, training, evaluation, deployment), use Kubeflow Pipelines or Argo Workflows. These run as Kubernetes CRDs and handle retries, caching, and artifact tracking.

Security and Compliance for AI

AI introduces unique security risks that go beyond traditional app security:

• Model Poisoning and Adversarial Attacks: Validate input data and monitor for anomalies. In production, use input filtering and rate limiting.
• Data Privacy: For regulated industries, ensure data residency and encryption in transit and at rest. Use Kubernetes Secrets for API keys and model credentials, but consider integrating with a vault (HashiCorp Vault) for rotation.
• Multi-tenancy: Use namespaces, network policies, and pod security admission to isolate teams. For GPU sharing, ensure that MIG partitions are isolated at the hardware level.
• Audit Logging: Enable Kubernetes audit logs and capture who accessed which model and when. For compliance, you may need to retain logs for years.
• SBOM and Image Scanning: Scan container images for vulnerabilities, but also scan model files for malicious code. Tools like Trivy can scan OCI artifacts, but for pickle files, use a dedicated scanner.

Cost Management and Optimization

AI workloads can be expensive due to GPU costs. Kubernetes gives you tools to control spend:

• Right-Sizing: Use VPA to recommend resource requests for CPU and memory. For GPUs, you must manually choose the right GPU type (e.g., A10 vs. A100) based on model size and throughput.
• Autoscaling: Use cluster autoscaler to scale nodes up/down based on pending pods. For spot instances, use node pools with spot instances for batch jobs, but set a budget.
• Cost Monitoring: Use tools like Kubecost or OpenCost to allocate costs by namespace, deployment, or label. This is essential for chargeback to business units.
• Efficient Inference: Use model compression and batching to reduce the number of GPUs needed. For LLMs, consider using smaller models (distillation) or quantization to fit on fewer GPUs.

Operational Readiness: Testing, Rollback, and Monitoring

Finally, you need to prove that your system can handle failures and updates without downtime:

• Chaos Engineering: Regularly kill pods, nodes, or simulate network partitions to test resilience. Use tools like Litmus or Chaos Mesh.
• Rolling Updates: Configure PodDisruptionBudgets (PDBs) to ensure availability during node maintenance. For model updates, use canary deployments and monitor error rates.
• Rollback Strategy: Keep previous model versions deployed and ready to roll back. Use a service mesh to shift traffic instantly.
• Performance Testing: Load test your inference endpoint with realistic traffic. Use tools like Locust or k6. Measure latency percentiles (p99) and throughput.
• Documentation: Document runbooks for common failures (GPU OOM, model drift, etc.). Ensure on-call engineers know how to access logs and metrics.

Frequently Asked Questions

Do I need Kubernetes for AI in production?

Not always. If you have a single model with modest traffic, a managed service (e.g., SageMaker, Vertex AI) might be simpler. Kubernetes is beneficial when you need portability across clouds, multi-tenant isolation, or fine-grained control over GPU scheduling.

How do I handle GPU memory oversubscription?

For CPU workloads, Kubernetes allows memory oversubscription, but for GPUs, it’s risky. Use MIG to partition GPUs, or set GPU memory limits via the device plugin. Monitor GPU memory usage and set alerts for high utilization.

What is the best way to serve large language models (LLMs) on Kubernetes?

For LLMs, use a specialized inference server like vLLM or TensorRT-LLM, which support continuous batching. Deploy them with autoscaling based on GPU utilization or queue length. Consider using a serving mesh to handle multi-model routing.

How do I ensure model versioning and reproducibility?

Store models in a registry with a unique hash. In Kubernetes, use immutable image tags and deploy via GitOps (Argo CD). For training, use a pipeline that records the dataset version, code commit, and hyperparameters.

What are the common mistakes when running AI on Kubernetes?

The most common mistakes are: not setting resource requests/limits for CPU and memory, ignoring GPU memory limits, using the default scheduler for distributed training, and not having a rollback plan for model updates. Also, forgetting to secure model artifacts.

How do I monitor model drift in production?

Use a monitoring system that compares prediction distributions over time. Tools like Evidently AI or Prometheus can track drift metrics. When drift is detected, trigger an alert and potentially rollback to a previous model.

Related

• NVMe Cache for Media NAS: What Gets Faster?
• Git Failed to Push Some Refs: Fixes That Work
• Why Evaluate AI Before Scaling an AI Curriculum?