Run AI Agents Safely with Kubernetes Controllers

This guide shows you how to run AI agents alongside Kubernetes controllers without compromising cluster stability or security. You’ll learn how to isolate agent workloads, control their resource usage, and prevent them from interfering with the control plane or other applications. This approach is essential for teams deploying autonomous agents that make API calls, process data, or take actions in response to events.

What You’ll Need

A Kubernetes cluster (v1.28 or later recommended) with RBAC enabled
kubectl configured with cluster-admin or namespace-admin privileges
Helm 3 (optional, for installing metrics-server or other tools)
Basic familiarity with YAML manifests and Kubernetes concepts (Pods, Deployments, Services)
Access to a container registry where your AI agent images are stored

Step-by-Step

Create a dedicated namespace for AI agents. Run kubectl create namespace ai-agents. This isolates agent workloads from your main applications and makes it easier to apply specific policies.

Define a ResourceQuota for the namespace. Create a YAML file (e.g., quota.yaml) with:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: agent-quota
  namespace: ai-agents
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
    pods: "10"

Apply it with kubectl apply -f quota.yaml. This prevents agents from consuming all cluster resources.

Set resource requests and limits on every agent container. In your agent Deployment spec, always include:
```
resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 1Gi
```
Adjust values based on your agent’s needs. Limits are critical to prevent memory leaks from crashing the node.

Apply a NetworkPolicy to restrict egress traffic. Create network-policy.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: agent-egress
  namespace: ai-agents
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except: ["169.254.169.254/32"]  # block cloud metadata
    ports:
    - port: 443
      protocol: TCP
    - port: 80
      protocol: TCP

This allows HTTPS/HTTP but blocks access to internal metadata endpoints. Adjust CIDRs to your needs.

Create a ServiceAccount with minimal RBAC permissions. Do not run agents with your personal credentials. Create agent-sa.yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: agent-sa
  namespace: ai-agents
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-role
  namespace: ai-agents
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments"]
  verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-binding
  namespace: ai-agents
subjects:
- kind: ServiceAccount
  name: agent-sa
  namespace: ai-agents
roleRef:
  kind: Role
  name: agent-role
  apiGroup: rbac.authorization.k8s.io

Apply it. Grant only the permissions the agent absolutely needs. If it doesn’t need to read cluster state, omit those rules.

Use a Pod Security Admission (PSA) label on the namespace. Run kubectl label namespace ai-agents pod-security.kubernetes.io/enforce=restricted. This forces agents to run with non-root, read-only root filesystem, and no privileged escalation, which is a baseline for safe execution.

Deploy your agent as a Deployment or StatefulSet. Example Deployment snippet:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-agent
  namespace: ai-agents
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-agent
  template:
    metadata:
      labels:
        app: my-agent
    spec:
      serviceAccountName: agent-sa
      containers:
      - name: agent
        image: myregistry/my-agent:latest
        resources:
          requests: { cpu: 100m, memory: 256Mi }
          limits: { cpu: 500m, memory: 1Gi }
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000

Apply it. Use kubectl rollout status deployment/my-agent -n ai-agents to check.

Set up a HorizontalPodAutoscaler (HPA) if your agent scales with load. Create hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-agent-hpa
  namespace: ai-agents
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-agent
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

Apply it. This ensures the agent scales without exceeding quota.

Monitor agent logs and metrics. Install Prometheus and Grafana (or use cloud monitoring). Create a ServiceMonitor for your agent pods. Regularly check kubectl logs -n ai-agents -l app=my-agent for unexpected behavior.

Implement a PodDisruptionBudget (PDB) for availability. Create pdb.yaml:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-agent-pdb
  namespace: ai-agents
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-agent

Apply it. This prevents voluntary disruptions (like node drains) from taking down all agent replicas.

Common Mistakes

Running agents with cluster-admin privileges. This is the most dangerous mistake. If the agent is compromised, it can delete any resource or exfiltrate secrets. Always use least-privilege RBAC.
No resource limits. Agents that make many API calls or process large data can consume unbounded memory, causing OOM kills or node pressure. Always set limits.
Allowing egress to cloud metadata endpoints. Agents can access IAM credentials if you don’t block 169.254.169.254. Use NetworkPolicies.
Running as root. A vulnerability in the agent code can lead to host compromise if run as root. Enforce restricted PSA.
Sharing namespaces with critical workloads. If an agent goes rogue, it can affect other apps. Use a dedicated namespace.

Tips

Use a separate node pool for AI agents if you have GPU or high-CPU requirements. This prevents noisy neighbors.
Set automountServiceAccountToken: false on the Pod spec if the agent doesn’t need to talk to the Kubernetes API. This reduces attack surface.
Use a read-only root filesystem and mount a temporary volume for any writes. Agents rarely need to write to the filesystem.
Implement a liveness probe that checks the agent’s internal health endpoint. This helps Kubernetes restart stuck agents.
Use a ConfigMap for agent configuration so you can change behavior without rebuilding the image.

Troubleshooting

Agent pods stuck in Pending. Check kubectl describe pod <pod-name> -n ai-agents. Likely causes: insufficient resources (quota exceeded), or the node selector doesn’t match any nodes. Adjust quota or nodeSelector.
Agent cannot connect to external APIs. Verify NetworkPolicy allows egress to the required IPs/ports. Use kubectl get networkpolicy -n ai-agents to check. Temporarily add a rule for testing.
Agent gets 403 Forbidden when calling Kubernetes API. Check the Role and RoleBinding. Ensure the ServiceAccount is correctly referenced and the verbs/resources match.
Agent keeps restarting with CrashLoopBackOff. Inspect logs: kubectl logs <pod-name> -n ai-agents --previous. Common issues: read-only filesystem (if you didn’t mount a writable volume), or missing environment variables.
HPA not scaling. Ensure metrics-server is installed and the agent has CPU requests set. Run kubectl get hpa -n ai-agents to see current metrics.

Frequently Asked Questions

Can I run an AI agent that needs to modify cluster resources?

Yes, but only if absolutely necessary. Grant specific RBAC permissions for the exact resources and verbs. For example, if the agent needs to scale a deployment, give it update on that specific deployment. Avoid wildcard permissions.

How do I prevent an agent from accessing secrets?

Do not mount secrets as environment variables or volumes unless required. Use RBAC to deny access to secrets. Also, ensure the agent’s ServiceAccount has no permissions to read secrets.

What if my agent needs GPU resources?

Add a resource request for nvidia.com/gpu (or your vendor’s GPU resource) and ensure the node has GPU drivers. Also, set a limit equal to the request. Be aware that GPU resources are not shareable, so plan accordingly.

Can I run multiple agents in the same namespace?

Yes, but consider using separate namespaces per agent or team to avoid interference. If they share a namespace, ensure each has its own ServiceAccount and RBAC.

How do I update an agent without downtime?

Use a rolling update strategy (default for Deployments). Set maxUnavailable: 0 and maxSurge: 1 to ensure at least one pod is always running. Also, update the PDB to require at least one available.

Is it safe to run agents that make outbound internet calls?

Yes, but restrict egress to only necessary domains and ports. Use NetworkPolicy with specific CIDRs or use an egress proxy. Avoid allowing all outbound traffic.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →