Watchdog Restarting Healthy Agent: Fix the Check

The core issue is that a watchdog (like systemd, supervisord, or a custom script) is restarting a process that appears healthy because its health check is misconfigured — often checking the wrong endpoint, using a too-short timeout, or failing to account for startup time. To fix it, you need to align the check with how the agent actually reports readiness and stability, and adjust the watchdog’s thresholds accordingly.

Background: What Watchdogs Do and Why They Restart

Watchdogs exist to keep services running. They monitor a process by sending signals (like SIGKILL) or by running a health check command. If the check fails, the watchdog assumes the process is hung or crashed and restarts it. This is great for truly dead services, but problematic when the check itself is flawed.

A “healthy agent� being restarted usually means the watchdog is interpreting a legitimate state as failure. Common scenarios include:

• The agent takes longer to start than the watchdog’s StartLimitInterval or TimeoutStartSec allows.
• The health check hits an endpoint that returns a non-200 status during normal operation (e.g., a maintenance mode).
• The check uses a timeout shorter than the agent’s response time under load.
• The watchdog checks a child process or a port that isn’t the agent’s main listener.

How the Watchdog Check Works (and Where It Fails)

Most watchdogs work in one of two ways:

1. Signal-based: The watchdog sends a signal (like SIGCHLD or a custom one) and expects a response within a timeout. If no response, it kills the process.
2. Command-based: The watchdog runs a command (like curl or a custom script) and checks the exit code. Non-zero means failure.

The failure usually lies in the command or the timeout. For example, a systemd service with WatchdogSec=30 expects the agent to call sd_notify(WATCHDOG=1) every 30 seconds. If the agent only sends that notification during startup or during heavy processing, the watchdog will think it’s dead. Similarly, a supervisord config with startsecs=10 will restart a process if it exits within 10 seconds of starting — but if the agent takes 15 seconds to initialize and then exits cleanly (e.g., to re-exec), it will be flagged.

How to Fix the Check: Step-by-Step

Here’s a systematic approach to stop the false restarts:

1. Identify the exact check. Look at the watchdog config (e.g., /etc/systemd/system/youragent.service or supervisord.conf). What command does it run? What timeout? What interval?
2. Test the check manually. Run the exact command as the same user. If it returns non-zero, you’ve found the problem. For example, curl -f http://localhost:8080/health might fail because the endpoint requires a specific header.
3. Adjust the timeout. If the agent takes 5 seconds to respond under load but the check times out at 2 seconds, increase it. In systemd, set TimeoutStartSec=30 or WatchdogSec=60. In supervisord, increase startsecs and stopwaitsecs.
4. Fix the endpoint. If the health endpoint returns 500 during startup or maintenance, either change the check to a different endpoint (like /ping) or make the agent return 200 even when not fully ready (but that defeats the purpose — better to have a dedicated liveness vs. readiness probe).
5. Use a custom script. If the built-in checks are too blunt, write a script that checks the agent’s actual state (e.g., a process is running and a file exists). Make sure the script exits 0 only when the agent is truly healthy.
6. Test the fix. After changing the config, restart the watchdog and observe. Use systemctl status or supervisorctl status to see if the restart count stops increasing.

Common Pitfalls and Misconceptions

One common mistake is confusing liveness with readiness. A liveness check asks “is the process alive?â€� — a simple kill -0 or a TCP connection. A readiness check asks “can it serve traffic?â€� — which requires the agent to have finished loading configs, connecting to databases, etc. If you use a readiness check for liveness, you’ll get false restarts.

Another pitfall is ignoring the exit code of the check command. For example, curl -f returns non-zero on HTTP errors, but curl without -f returns zero even for a 500 response. If your check doesn’t use -f, a 500 won’t trigger a restart — which is good if the agent is healthy, but bad if the agent is actually down.

Also, watch out for timezone or clock skew issues if the watchdog uses timestamps. Rare, but possible.

Comparing Watchdog Configurations

Different watchdogs have different defaults. Here’s a quick comparison:

• systemd: Uses WatchdogSec for hardware watchdog and Restart=on-failure with RestartSec for service restarts. It’s powerful but requires the agent to support sd_notify for true watchdog integration.
• supervisord: Uses startsecs (how long the process must stay up to be considered started) and startretries. It’s simpler but less flexible for health checks — you often need a custom healthcheck script.
• Docker/Kubernetes: Uses livenessProbe and readinessProbe. The liveness probe is the watchdog; the readiness probe controls traffic. If you set the liveness probe to the readiness endpoint, you’ll get restarts during startup.

For a custom script, you have full control. But you must ensure the script itself is fast and reliable — a slow script can cause the watchdog to think the check is hung.

Costs and Trade-offs of Fixing the Check

Fixing the check isn’t free. You might need to:

• Add a dedicated health endpoint to your agent (development time).
• Write and maintain a custom check script.
• Increase timeouts, which means the watchdog will take longer to detect a real failure.

There’s a trade-off between false positives (restarting a healthy agent) and false negatives (not restarting a dead one). A too-lenient check might let a truly hung agent run indefinitely. A too-strict check causes churn. The right balance depends on your tolerance for downtime vs. unnecessary restarts.

Safety and Best Practices

When modifying watchdog configs, always test in a staging environment first. If you’re using systemd, make a backup of the service file and use systemd-analyze verify to check for syntax errors. For supervisord, use supervisorctl reread and supervisorctl update to apply changes without restarting everything.

Also, consider adding logging: have the watchdog log why it restarted the process. In systemd, you can see the last restart reason in journalctl -u youragent. In supervisord, check the log file for the process. This will help you diagnose future issues.

Frequently Asked Questions

Why does my agent get restarted even though it’s running fine?

Most likely the health check is failing due to a timeout or a wrong endpoint. Run the check command manually to see the exit code. If it’s non-zero, adjust the check or the timeout.

What’s the difference between liveness and readiness probes?

Liveness probes determine if the process is alive and should be restarted if it fails. Readiness probes determine if the process can serve traffic and should be removed from load balancers if it fails. Using a readiness probe as a liveness probe will cause restarts during startup or maintenance.

How do I set a custom health check in systemd?

You can use ExecStartPre or a separate ExecStartPost script, but for continuous checking, you’re better off using WatchdogSec with the agent calling sd_notify. Alternatively, use a timer unit that runs a script and restarts the service on failure.

What should I do if increasing the timeout doesn’t fix it?

Check if the agent is actually crashing and restarting on its own. Look at the agent’s logs for errors. If it’s crashing, the watchdog is doing its job — you need to fix the agent, not the check.

Can I use a TCP check instead of an HTTP check?

Yes, a TCP check (like nc -z localhost 8080) only verifies that a port is open. That’s good for liveness but not for readiness. For a more accurate check, use HTTP or a custom script.

Is it safe to disable the watchdog temporarily?

Yes, for debugging. In systemd, run systemctl stop youragent.service and then start it manually. In supervisord, use supervisorctl stop youragent. Just remember to re-enable it after fixing the issue.

Related

• What to Check Before Buying the Dip in Indian Shares
• Cool Raspberry Pi Projects: 15 Ideas for Any Skill Level
• Terraform vs Kubernetes: Key Differences