When a Kubernetes node fails, two different components can detect it: the cloud controller manager (CCM) and the node problem detector (NPD). The cloud controller manager detects failures from the cloud provider’s perspective (e.g., the VM is gone), while the node problem detector detects failures from inside the node (e.g., the kubelet is hung or the disk is full). You need both, because they catch different types of failures and trigger different remediation actions.
Formula
The calculator above estimates the ratio between the time a cloud controller manager takes to detect a node failure versus the time the node problem detector takes. The formula is:
Detection ratio = CCM detection delay / NPD detection delay
Where:
- CCM detection delay (t) – the total time (in seconds) you expect the cloud controller to take, including the cloud provider API polling interval and any health check timeouts.
- Cloud provider check interval (p) – how often the CCM polls the cloud provider for node status (in seconds).
- NPD check interval (n) – how often the node problem detector checks local conditions (in seconds).
This ratio helps you understand which component will likely flag a failure first. A ratio above 1 means the NPD will detect the issue sooner; below 1 means the CCM will.
Worked Example
Using the default values from the calculator:
- CCM detection delay = 30 seconds
- Cloud provider check interval = 10 seconds
- NPD check interval = 5 seconds
Calculate the denominator: 10 + 5 = 15 seconds.
Now divide: 30 / 15 = 2.0.
The result is 2.0, meaning the CCM is expected to take twice as long as the NPD to detect a node failure. This matches the calculator output.
Background: Why Two Detectors?
Kubernetes is designed to run across many machines, and nodes can fail in many ways. A node might be powered off, lose network connectivity, or have a kernel panic. The control plane needs to know when a node is no longer usable so it can reschedule pods and update service endpoints.
The cloud controller manager is the component that talks to your cloud provider (AWS, GCP, Azure, etc.). It checks whether the underlying VM or instance is still running. If the instance is terminated, the CCM marks the node as NotReady and eventually deletes it from the cluster.
The node problem detector is a daemon that runs on each node. It monitors local conditions: kernel logs, file system capacity, kubelet responsiveness, and other system-level issues. When it finds a problem, it reports it as a condition on the node object, which can then trigger automated responses.
How the Cloud Controller Manager Detects Failures
The CCM works by periodically querying the cloud provider’s API for the status of the instances that back your nodes. The exact mechanism varies by provider, but generally:
- The CCM has a reconciliation loop that runs every few seconds (often configurable via
--node-monitor-period). - It checks if the instance is running, stopped, or terminated.
- If the instance is stopped or terminated, the CCM updates the node’s
Readycondition toUnknownorFalse. - After a timeout (default
--node-monitor-grace-periodis 40 seconds), the node is markedNotReady. - If the node remains unreachable for a longer period (default
--pod-eviction-timeoutis 5 minutes), pods are evicted and the node is eventually deleted.
The key advantage of the CCM is that it detects failures that are invisible from inside the node – for example, if the VM is terminated by the cloud provider or the network is partitioned at the hypervisor level.
How the Node Problem Detector Works
The node problem detector runs as a DaemonSet on every node. It uses a set of monitors to check local conditions:
- Kernel monitor – watches the kernel log for oops, hung tasks, or other critical errors.
- System log monitor – checks systemd journal for failures like disk errors or service crashes.
- Custom plugin monitor – lets you define your own checks using shell commands or HTTP endpoints.
When a problem is detected, the NPD updates the node’s conditions (e.g., KernelDeadlock, ReadonlyFilesystem, FrequentUnregisterNetDevice). It can also send events to the API server. The key difference is that the NPD detects problems that do not necessarily make the node unreachable – the node is still running, but it is unhealthy.
Comparing the Two: What Each Catches
Here is a practical comparison of what each detector catches:
- CCM catches: instance termination, network partition at the cloud level, hypervisor failure, and any scenario where the VM disappears.
- NPD catches: kernel deadlock, disk full, read-only file system, kubelet hang, and other local resource exhaustion.
There is an overlap: if the node’s network is down, both may detect it – the CCM because it cannot reach the instance, and the NPD because the kubelet cannot reach the API server. But they are not redundant because many failures are only visible from one side.
Common Problems and Misconceptions
A common misconception is that the CCM alone is sufficient. In reality, a node can be perfectly healthy from the cloud provider’s perspective (the VM is running) but be completely unusable due to a kernel panic or a full disk. The CCM will never mark it as failed, so pods will keep running on a broken node.
Another issue is that the NPD can produce false positives. For example, a temporary spike in disk usage might trigger a ReadonlyFilesystem condition, causing pods to be evicted unnecessarily. Tuning the NPD’s thresholds and check intervals is important.
Also, the CCM’s detection is only as fast as the cloud provider’s API. Some providers have slow API responses or rate limiting, which can delay detection. The NPD, running locally, is usually faster for local issues.
How to Choose and Configure
In practice, you should run both. The CCM is a required component for any cloud-based Kubernetes cluster (it is part of the control plane). The NPD is optional but highly recommended. Here is how to configure them:
- Set the CCM’s
--node-monitor-grace-periodto a value that balances speed with false positives – 40 seconds is the default, but you can lower it if you want faster detection. - Configure the NPD’s monitors to match your workloads. For example, if you run stateful databases, you might want to watch for disk I/O errors.
- Use the NPD’s
--system-log-monitorwith a custom config to catch issues specific to your OS. - Consider using the NPD’s
--enable-rebootflag to automatically reboot a node if a fatal condition is detected.
You can also set up alerts based on the conditions each detector reports. For instance, you might want an alert when the CCM marks a node NotReady, and a different alert when the NPD reports a kernel deadlock.
Frequently Asked Questions
Can the node problem detector replace the cloud controller manager?
No. The NPD cannot detect failures that occur at the cloud provider level, such as the VM being terminated. The CCM is the only component that can detect that. They serve different purposes and are complementary.
What happens when both detect a failure at the same time?
Both will update the node’s conditions. The CCM will set Ready to False, and the NPD will set a specific condition like KernelDeadlock. The kubelet and scheduler will react to the most severe condition. The node will be marked NotReady, and pods will be rescheduled after the eviction timeout.
How fast should my detection be?
It depends on your application’s tolerance for downtime. For critical workloads, you might want detection within 10-20 seconds. For less critical workloads, 1-2 minutes is usually acceptable. The CCM’s default grace period is 40 seconds, and the NPD’s default check interval is 5 seconds.
Does the node problem detector work on bare metal?
Yes. The NPD is not cloud-specific. It runs on any Linux node and monitors local conditions. The CCM, however, is cloud-specific – on bare metal, you would use a different controller or rely on the kubelet’s own node status updates.
Can I use the NPD to detect network issues?
Not directly. The NPD monitors local conditions, not network connectivity. For network issues, you would need a separate tool like a network checker or rely on the CCM’s ability to detect unreachable instances.