Node.js SaaS Monitoring: 3 Key Failure Signals

Node.js SaaS monitoring should focus on three failure signals above all others: event loop lag, unhandled promise rejections, and memory leak trends. These three indicators directly reflect the health of your Node.js process and are the most likely to cause user-facing outages if ignored. This article explains why these signals matter, how to detect them, and what to do when they appear.

Why These Three Signals Matter Most

Node.js is single-threaded for JavaScript execution. That means one blocked event loop stops all request handling, no matter how many CPU cores your server has. Unhandled promise rejections are a leading cause of crashes in modern Node.js applications, especially after the default behavior changed in Node 15 to crash the process. Memory leaks are insidious because they degrade performance slowly, eventually leading to out-of-memory kills that take down your entire service.

Other metrics like CPU usage, disk I/O, or network latency are important, but they are often symptoms of underlying issues. High CPU may be a runaway loop or a memory leak causing constant garbage collection. High latency might be caused by a slow database, but it could also be event loop lag. The three signals above are the most direct indicators of Node.js-specific failure modes.

What Is Event Loop Lag and How to Measure It

The event loop is the core of Node.js. It processes callbacks, timers, and I/O events. When a synchronous operation takes too long, the event loop cannot process new events, and all requests wait. Event loop lag is the delay between when an event is ready and when it is actually processed.

To measure it, you can use Node’s built-in process.hrtime() to measure the delay of a setTimeout call. A simple polling script:

const start = process.hrtime.bigint();
setInterval(() => {
  const now = process.hrtime.bigint();
  const lag = Number(now - start) / 1e6; // milliseconds
  console.log(`Event loop lag: ${lag.toFixed(2)} ms`);
  start = now;
}, 1000);

If lag is consistently above 100 ms, your event loop is blocked. Above 500 ms, users will notice delays. Above 1 second, your service is effectively down.

Common causes include CPU-heavy synchronous operations, JSON parsing of huge payloads, complex regex, or infinite loops. Fixing lag often means moving heavy work to worker threads, breaking up large tasks, or using streams instead of buffering entire files.

Unhandled Promise Rejections: The Silent Killer

In Node.js, a promise rejection that is not caught triggers the unhandledRejection event. Since Node 15, the default behavior is to crash the process, turning a bug into an outage. Before that, it only printed a warning, letting errors go unnoticed.

To monitor this, you can attach a handler:

process.on('unhandledRejection', (reason, promise) => {
  // Log and alert, but do not exit immediately
  console.error('Unhandled rejection at:', promise, 'reason:', reason);
  // In production, you might want to exit and let the process manager restart
});

But monitoring the event is not enough. You need to know the stack trace and the context. Use a logging library that captures the error object, the promise, and any relevant request IDs. Set up alerts so that any unhandled rejection triggers a notification.

The best fix is to always attach a .catch() to every promise. Linting rules like no-floating-promises in TypeScript can help enforce this at build time. Also, consider using process.on('multipleResolves') to catch cases where a promise is resolved multiple times, which can also indicate a logic bug.

Memory Leak Trends: The Slow Burn

Memory leaks in Node.js often come from global variables, cached data without eviction, closures, or listeners that are never removed. The process’s memory usage slowly grows until it hits the heap limit (default around 2 GB on 64-bit systems) and crashes with heap out of memory.

Monitoring memory is not just about the current value. You need to track the trend over time. A healthy process will show a sawtooth pattern: memory rises as objects are allocated, then drops after garbage collection. A leak shows a steady upward trend with no recovery.

Use process.memoryUsage() to get heapUsed and heapTotal. Collect this data every minute and store it in your monitoring system. Look for a slope that is consistently positive over hours or days. Set an alert if the heap usage grows by more than, say, 10% per hour over a 6-hour window.

To debug a leak, take heap snapshots using the v8.writeHeapSnapshot() method or the Chrome DevTools memory profiler. Compare snapshots at different times to see which objects are growing. Common culprits include:

• Unbounded caches (e.g., using a Map without a size limit)
• Event listeners added in a loop without removal
• Closures that reference large objects
• Streams that are not properly closed

How to Monitor These Signals in Production

You can use any APM or monitoring tool that supports Node.js. The key is to collect the metrics with a custom script and send them to your dashboard. For example, using the prom-client library for Prometheus:

const client = require('prom-client');
const gauge = new client.Gauge({ name: 'event_loop_lag', help: 'Event loop lag in ms' });
setInterval(() => {
  const lag = measureLag(); // your function
  gauge.set(lag);
}, 1000);

Set up alerts with thresholds. For event loop lag, alert if it exceeds 200 ms for more than 1 minute. For unhandled rejections, alert immediately. For memory, alert if the heap used exceeds 80% of the heap limit, or if the trend is upward for 6 hours.

Also monitor the process’s uptime and restarts. Frequent restarts may indicate crashes from unhandled rejections or OOM kills. In a containerized environment, check if the container is being killed by the kernel OOM killer.

Common Mistakes and Misconceptions

One common mistake is only monitoring CPU usage. High CPU can be caused by a memory leak, so you might fix the wrong thing. Another mistake is ignoring unhandled rejections because “the app still works” – but it may crash at any moment.

Another misconception is that event loop lag is only caused by synchronous code. It can also be caused by too many timers or I/O operations that are not properly async. For example, using fs.readFileSync inside a request handler blocks the loop.

People often forget to monitor the unhandledRejection event in production because they only test with --unhandled-rejections=strict locally. But in production, the default is to crash, so you need to have a process manager like PM2 or systemd to restart the app automatically and alert you.

How to Choose a Monitoring Solution

When choosing a monitoring tool, look for one that supports custom metrics and alerting. Many APMs like New Relic or Datadog have built-in Node.js metrics, but you may need to send custom metrics for event loop lag and unhandled rejections. Open-source options like Prometheus + Grafana are flexible and free.

Consider the following factors:

• Ease of integration with your existing stack
• Alerting capabilities (email, Slack, PagerDuty)
• Histogram and trend analysis for memory
• Ability to track custom events like unhandled rejections
• Cost, especially for high-volume metrics

Also, decide whether you want to monitor at the process level or the cluster level. If you run multiple instances, you need to aggregate metrics across all of them.

Frequently Asked Questions

What is a safe event loop lag threshold?

For most web applications, a lag of under 50 ms is fine. Above 100 ms, users may notice delays. Above 500 ms, your service is likely timing out requests. Set your alert at 100 ms for early warning.

Should I crash the process on unhandled rejection?

In production, it is safer to crash and restart than to continue with a broken state. But you should log the error first. Use a process manager to restart automatically and alert you. The goal is to fail fast and recover quickly.

How can I detect a memory leak without a monitoring tool?

You can use the built-in v8.writeHeapSnapshot() to take snapshots and compare them. Also, watch the heapUsed in process.memoryUsage() over time. If it never drops after garbage collection, you likely have a leak.

Can I use these metrics for a multi-instance deployment?

Yes, you should aggregate them. Use a monitoring system that can sum or average metrics across instances. For event loop lag, take the maximum across instances, because one slow instance affects users routed to it.

What is the difference between heapUsed and rss?

heapUsed is the memory used by V8 for JavaScript objects. rss (resident set size) is the total memory the process holds, including native memory and buffers. For Node.js-specific leaks, focus on heapUsed. For native leaks, watch rss.

How often should I check these metrics?

Collect them every 10-30 seconds for real-time monitoring, and store historical data for trend analysis. Alerting should be based on sustained thresholds, not single spikes, to avoid false alarms.

Related

• Auto Mute & Camera Off on Google Meet
• Best RAM for Gaming: How to Choose
• Tesla Phone Charger: Correct Way to Use It