Kubernetes API Server Throttling & Saturation Fix

This guide walks you through diagnosing and resolving Kubernetes API server throttling and request saturation. You’ll learn how to identify the root cause—whether it’s client-side rate limiting, server-side overload, or resource exhaustion—and apply targeted fixes to restore responsiveness. You’ll need kubectl access to your cluster and the ability to view logs and metrics.

What You’ll Need

• Cluster administrator access (to view kube-apiserver logs and modify flags)
• kubectl configured with sufficient permissions
• Access to cluster metrics (e.g., Prometheus, Cloud Provider metrics, or `kubectl top`)—optional but helpful
• Basic understanding of Kubernetes control plane components

Step-by-Step

1. Confirm the symptom. Users or controllers report errors like 429 Too Many Requests, Timeout, or etcdserver: request timed out. Check if the API server is slow to respond or if requests are being rejected. Use kubectl get --raw /healthz to check basic health.
2. Check API server metrics. Access the API server metrics endpoint (usually /metrics). Look for apiserver_request_total, apiserver_request_duration_seconds, and especially apiserver_current_inflight_requests. The inflight requests metric shows how many requests are currently being processed. If it’s near the configured limit (default --max-requests-inflight is 400 for read requests and --max-mutating-requests-inflight is 200 for write requests), you’re hitting saturation.
3. Look for client-side throttling. If you see Throttling request took 1.234s in your client logs (kubectl, controller-manager, scheduler), that indicates client-side rate limiting. Check the kubeconfig’s qps and burst settings. The default for kubectl is qps=5, burst=10, which is very low. For controllers, the default is often qps=20, burst=30. Increase these values if appropriate.
4. Identify the source of high request volume. Use the apiserver_request_total metric, broken down by verb, resource, and client. A common culprit is a misconfigured controller or a custom controller with a tight watch loop. Also check if there are many LIST calls with large resource versions—these are expensive.
5. Check etcd health and performance. Since the API server stores all data in etcd, etcd latency directly affects API server performance. Check etcd metrics (if exposed) or run etcdctl endpoint health from the etcd pod. Look for high disk I/O, network latency, or slow fsync. If etcd is slow, the API server will time out even if it has capacity.
6. Review API server flags. Check the API server’s startup flags. Look at --max-requests-inflight and --max-mutating-requests-inflight. If they’re set too low for your workload, you’ll see throttling. Also check --request-timeout (default 60s). If requests are timing out, consider increasing it, but first understand why they’re slow.
7. Inspect API server logs. Look for patterns like Too many requests, APIService errors, or etcdserver: request timed out. Also look for WATCH events that are being dropped or taking too long. Use kubectl logs -n kube-system kube-apiserver-.
8. Check for watch leaks. A common cause of saturation is too many open watch connections. Each watcher consumes resources. Use the apiserver_watch_events_sizes or apiserver_watch_events_total metrics to see watch activity. If a client is not closing watches properly, it can exhaust the server.
9. Scale the API server. If you’re running a single API server, consider running multiple replicas (typically 2-3). This spreads the load. Ensure your load balancer distributes traffic evenly. Also consider increasing the node size if CPU/memory are saturated.
10. Optimize client behavior. For controllers, use informers with proper resync periods instead of polling. For kubectl, use --server-side for large apply operations. For custom scripts, batch requests and use watch instead of frequent GETs.
11. Implement priority and fairness. If you’re on Kubernetes 1.20+, enable the APIPriorityAndFairness feature gate (it’s beta by default). This allows you to configure FlowSchema and PriorityLevelConfiguration to prevent a single client from starving others. Create a flow schema that limits aggressive clients.
12. Consider API caching. For read-heavy workloads, use a cache like the kube-aggregator or a sidecar cache. This reduces direct load on the API server.

Common Mistakes

• Ignoring client-side throttling. Many users see 429 errors and assume the server is overloaded, but the client is actually rate-limiting itself. Always check client logs first.
• Setting max-requests-inflight too high. Increasing this blindly can cause memory exhaustion and crash the API server. It should be tuned based on available memory and typical request size.
• Not monitoring etcd. The API server is only as fast as etcd. If etcd is slow, no amount of API server tuning will fix it.
• Forgetting about watch requests. Watches are long-lived and can consume significant memory. A single client opening thousands of watches can be worse than thousands of GETs.

Tips

• Use kubectl get --raw /metrics | grep apiserver_current_inflight to quickly check current saturation.
• Enable audit logging temporarily to see which clients are making the most requests.
• For production, set up Prometheus alerts on apiserver_current_inflight_requests exceeding 80% of the limit.
• When increasing client QPS, also increase burst to allow for short spikes.
• Use the --v=4 flag on the API server to get more detailed request logs, but be cautious as this increases log volume.

Troubleshooting

If the above steps don’t resolve the issue, try the following:

• Check if the issue is intermittent. If it happens only during certain times, it may be related to a cron job or a specific workload. Use metrics to correlate.
• Verify network connectivity. High latency between clients and the API server can cause timeouts. Use kubectl get --raw /readyz and check response times.
• Review recent changes. If the problem started after a change, roll back or adjust the change. Check for new controllers or updated versions.
• Check for resource leaks. If memory usage of the API server keeps increasing, there may be a leak (often due to watches). Restart the API server as a temporary measure, but investigate the root cause.
• If all else fails, restart the API server. This clears all transient state. It’s safe to do one at a time if you have multiple replicas.

Frequently Asked Questions

What is the difference between throttling and saturation?

Throttling is when the API server deliberately rejects or delays requests to protect itself (either via client-side rate limiting or server-side max-inflight limits). Saturation is when the API server is processing so many requests that it can’t respond in time, often due to CPU or etcd bottlenecks.

How do I increase the API server’s request limits?

You can increase the --max-requests-inflight and --max-mutating-requests-inflight flags on the kube-apiserver. However, be cautious—these limits exist to prevent memory exhaustion. Monitor memory usage after increasing.

Why do I see ‘Throttling request took 1.234s’ in my logs?

This is a client-side message, meaning your client (kubectl, controller) is rate-limiting itself. Check the qps and burst settings in your kubeconfig or client configuration.

Can a single namespace cause API server saturation?

Yes, if a namespace has many resources and many controllers watching them, it can generate a high volume of requests. Use priority and fairness to limit the impact of such namespaces.

Is it safe to restart the API server?

Yes, the API server is stateless (it stores data in etcd). Restarting it will briefly interrupt API access, but it will recover. If you have multiple replicas, restart them one by one to avoid downtime.

What is APIPriorityAndFairness?

It’s a feature that allows you to configure how requests are prioritized and limited. It prevents a single client from consuming all API server resources, ensuring fair access for all.

Related

• Slack Etiquette Tips: Better for Everyone?
• How to Become a VR Developer: A Practical Roadmap
• Troubleshoot TV No Signal Issue: Quick Fixes