keepcalculator

06 Numbers and Powers

Security Monitoring: Why More Data Still Misses Threats

Card No. 06·84688 · Updated · Standard formula, shown below

Security monitoring collects more data than ever, yet breaches still slip through because volume alone does not equal visibility. Adding more logs, alerts, and sensors often creates noise that hides the few signals that matter, and the real problem is not data collection but data interpretation, correlation, and response. This article explains why security teams drown in data while attackers go unnoticed, and what actually improves detection rates.

The Data Deluge: More Logs, More Blind Spots

Modern enterprises generate terabytes of logs daily from endpoints, cloud services, network devices, applications, and identity providers. The instinctive response to a breach is to collect more—more logs, longer retention, more telemetry. But this approach has a fundamental flaw: detection quality is not proportional to data volume.

When you add a new data source, you also add its inherent noise. A typical SIEM receives thousands of events per second, and the vast majority are routine—successful logins, normal network traffic, scheduled tasks. The signal-to-noise ratio is often 1 in 10,000 or worse. Analysts must wade through this to find the one anomaly that matters, and fatigue sets in quickly.

A concrete example: a company ingests firewall logs, DNS logs, and Windows Event Logs. An attacker uses a legitimate admin account to access a file share at 3 AM. That single event is buried among 50,000 other events from that hour. Unless the monitoring system has a baseline for “normal” behavior for that user, the event looks identical to a routine action. More data without context is just more noise.

Alert Fatigue and the Boy Who Cried Wolf

Alert fatigue is the most direct consequence of high data volume. When analysts see hundreds of alerts per day and 99% are false positives, they begin to ignore them. This is a well-documented phenomenon: the more alerts a system generates, the less likely any single one is to be investigated thoroughly.

The math is brutal. If a security operations center (SOC) has 5 analysts and receives 1,000 alerts per day, each analyst must triage 200 alerts daily. At even 2 minutes per alert, that is over 6 hours of pure triage—leaving no time for hunting, investigation, or response. The result is that alerts are skimmed, not analyzed, and critical ones are missed.

Moreover, false positives train analysts to distrust the system. When an alert says “possible malware” but turns out to be a software update, the next time that alert fires, the analyst may dismiss it without looking. Attackers exploit this by blending in with normal activity or by triggering a flood of low-severity alerts to distract from their real attack.

Correlation and Context: What More Data Doesn’t Give You

Data is just raw material; it becomes intelligence only when correlated with context. A single login event from a foreign IP address is meaningless unless you know whether that user typically travels, whether the IP is a known proxy, and whether the account has recently changed passwords. More data doesn’t provide this context—it requires enrichment and correlation logic.

Most monitoring tools rely on signature-based detection or simple threshold rules. They look for known attack patterns or unusual volumes. But modern attacks are designed to evade these rules. They use legitimate tools, slow and low movements, and encrypted traffic. The only way to catch them is to build a baseline of normal behavior for each user, device, and application, then detect deviations—a process called User and Entity Behavior Analytics (UEBA).

UEBA requires not just more data, but the right data and the ability to join it. For example, to detect a compromised account, you need to combine authentication logs, geolocation data, endpoint activity, and perhaps even HR records (to know if the user is on leave). Many organizations collect all these but store them in separate silos that never talk to each other. The data exists, but the correlation doesn’t.

Detection Gaps: What You’re Not Collecting

Ironically, while most organizations collect too much of some data, they collect too little of the data that matters. Common gaps include:

  • Cloud API logs – many cloud providers log API calls, but teams often don’t enable or retain them.
  • Email gateway metadata – the source, subject, and attachments of inbound email are often only stored for 30 days.
  • DNS queries – DNS is a common exfiltration channel, but many organizations don’t log it at all.
  • Endpoint process execution – Windows Event ID 4688 (process creation) is often disabled due to performance concerns.
  • Network flow data – full packet capture is rare; NetFlow or IPFIX summaries are more common but lack content.

Attackers know these gaps. They use PowerShell scripts that don’t trigger file-based signatures, or they use DNS tunneling to exfiltrate data because DNS logs are not reviewed. More data doesn’t help if the data that captures the attack is never collected.

How to Actually Improve Detection: Quality Over Quantity

The solution is not to collect less, but to collect smarter. Here are evidence-based approaches that work:

  1. Prioritize data sources – focus on the 20% of sources that provide 80% of detection value: authentication logs, endpoint process execution, DNS, and cloud API logs.
  2. Implement alert deduplication and aggregation – group similar alerts into incidents to reduce volume.
  3. Use risk-based alerting – assign a risk score to each alert based on asset criticality, user privilege, and attack phase, and only page on high-risk alerts.
  4. Build behavioral baselines – use machine learning to establish what is normal for each user and device, then alert on deviations.
  5. Invest in SOAR (Security Orchestration, Automation, and Response) – automate the triage of low-confidence alerts so analysts can focus on high-value investigations.
  6. Regularly tune detection rules – review false positive rates monthly and adjust thresholds or disable ineffective rules.

One practical example: instead of alerting on every failed login, alert only when the number of failed logins for a single account exceeds a threshold and is followed by a successful login from a different IP. This reduces volume while catching brute-force attacks that actually succeed.

Costs and Trade-offs of More Data

Storing and processing security data is expensive. SIEM licensing is often based on data volume per day, and costs scale linearly. A typical enterprise might spend hundreds of thousands of dollars per year on SIEM ingestion alone. Retaining data for 12 months (required by many compliance frameworks) multiplies storage costs.

But the hidden cost is human time. Every additional data source requires integration, parsing, and tuning. Analysts must learn the new schema, and detection engineers must write and test new rules. These costs are rarely accounted for when a team decides to “collect everything.”

There is also a performance trade-off. Collecting verbose logs can slow down endpoints and network devices. For example, enabling full process command-line logging on Windows can increase CPU usage by 5-10% on busy servers. The security benefit must be weighed against operational impact.

Frequently Asked Questions

Why do we miss attacks even with a SIEM?

Most SIEMs are rule-based and only detect known patterns. They miss novel attacks, and they generate so many false positives that analysts become desensitized. The SIEM is only as good as the rules and the data feeding it—if either is incomplete, attacks slip through.

Is more data ever the answer?

Yes, but only when it fills a specific gap. For example, if you don’t collect DNS logs, adding them can reveal command-and-control traffic. However, adding more of the same type of data you already have rarely helps. The key is to identify what attack techniques you can’t currently see, then collect the data that would reveal them.

What is the difference between a false positive and a true positive?

A true positive is an alert that correctly identifies malicious activity. A false positive is an alert that fires on benign activity—for example, a legitimate user logging in from a new location. The ratio between them determines your alert precision, and most teams have far more false positives than true positives.

How can we reduce alert fatigue without missing real threats?

Start by measuring your true positive rate and setting a goal to improve it. Use alert aggregation to group related events, risk-score every alert, and automate the investigation of low-risk alerts. Also, involve threat intelligence to prioritize alerts that match known adversary tactics, techniques, and procedures (TTPs).

Does machine learning solve the problem?

Machine learning helps by baselining behavior and detecting anomalies, but it is not a silver bullet. ML models require clean training data, can produce false positives themselves, and can be evaded by attackers who slowly adapt their behavior. The best results come from combining ML with human expertise and solid detection engineering.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →