Log Splitting by Business Key: A Resource Problem

Splitting logs by business key becomes a resource management problem because the number of log streams grows with the number of unique key values, and each stream consumes memory, file handles, and CPU for indexing and rotation. When the key has high cardinality (e.g., user IDs, order IDs, device IDs), you can end up with millions of tiny files or in-memory buffers, causing excessive overhead, slow queries, and eventually system instability.

Background: Why Logs Are Split

Log splitting (also called log partitioning or sharding) is a common practice to organize log data by a business dimension—customer, tenant, region, or transaction ID. The goal is to isolate one entity’s logs for easier troubleshooting, compliance, or billing. For example, a multi-tenant SaaS platform might split logs by tenant ID so that support can quickly pull all logs for a specific customer. Splitting also allows you to set different retention policies per business unit (e.g., keep audit logs for 7 years, debug logs for 30 days).

The problem is that the number of log streams equals the number of distinct key values. If you have 10,000 customers, you have up to 10,000 active log files or buffers. Each of those needs its own file descriptor, rotation timer, and index entry. When the cardinality is low (like 3 regions), this is trivial. When it’s high (like millions of user IDs), the overhead becomes a dominant cost.

How Log Splitting Works

In a typical logging pipeline, an application writes a log record that includes a business key (e.g., customer_id=12345). The logging agent (like Fluentd, Logstash, or a custom writer) reads that key and routes the record to a destination named after the key. This can be:

• File-based: Each key gets its own file, e.g., /logs/customer/12345.log.
• Buffer-based: Each key gets an in-memory buffer that is flushed periodically to a central store.
• Topic/stream-based: Each key maps to a separate Kafka topic or S3 prefix.

Each destination has its own lifecycle: open, write, rotate, close, delete. The logging system must track the state of every active stream, which includes the current file offset, the last rotation time, and the buffer size. This metadata is stored in memory and occasionally on disk.

The Core Resource Problem

The fundamental issue is that resources per stream are fixed, but the number of streams is variable and often unbounded. Let’s break down the costs:

• Memory: Each stream holds a buffer (typically 1–10 MB) and metadata (~100–500 bytes). With 100,000 streams, that’s 100 GB of RAM just for buffers, even if most streams are idle.
• File descriptors: Each open file consumes a file descriptor. Linux default limits are often 1024 per process (soft limit) and can be raised, but with 10,000 files, you’ll hit limits and need to tune the OS.
• CPU: Every log write requires a lookup to find the correct stream (hash or map lookup). With millions of keys, the lookup table becomes large, and cache misses increase latency.
• Disk I/O: Many small files cause inefficient writes. Filesystems have a block size (e.g., 4 KB), and writing 100 bytes to a file still consumes 4 KB. Also, rotating 1 million files at midnight causes a massive I/O spike.

In practice, the system spends more time managing streams than processing log data. This is why splitting by a high-cardinality key is often called a “log explosion” or “file descriptor exhaustion.”

Common Problems and Symptoms

When log splitting by business key goes wrong, you’ll see these symptoms:

• High memory usage: The logging agent’s RSS grows linearly with the number of distinct keys, even if the total log volume is constant.
• Too many open files: The process hits EMFILE (too many open files) and starts dropping logs or crashing.
• Slow startup: On restart, the agent must scan and reopen thousands of files, delaying log ingestion.
• Slow queries: Searching across many small files is slower than searching a few large files because of the overhead of opening and reading many inodes.
• Rotation storms: When all streams rotate at the same time (daily), the system tries to rename and compress thousands of files simultaneously, causing CPU and I/O spikes.

How to Choose a Splitting Strategy

Not all log splitting is bad. The key is to match the strategy to the cardinality and query pattern:

• Low cardinality (e.g., < 100): Splitting by business key is fine. Use one file per key, and you won’t notice overhead.
• Medium cardinality (100–10,000): Consider splitting by a coarser key (e.g., region instead of user) or using a single file with an indexed field (e.g., JSON logs with a key field).
• High cardinality (>10,000): Avoid splitting by the raw key. Instead, write all logs to a shared stream and add the key as a structured field. Use a search index (like Elasticsearch) to filter by key at query time.

Another approach is to use a two-tier system: write to a shared file, then have a background job that splits into per-key files only for keys that are actively queried (e.g., recent orders). This gives the benefit of isolation without the overhead for inactive keys.

Costs and Trade-offs

There’s always a trade-off between query convenience and resource usage. Splitting by key makes point queries fast—you only read one file. But it makes aggregate queries (e.g., “all logs for all customers in Europe”) slow because you must scan many files. Conversely, a single file with an index makes point queries slightly slower (index lookup) but aggregate queries fast.

Storage cost also increases with splitting because of filesystem block overhead and duplicate metadata. For example, a 1 KB log record stored in a file with a 4 KB block size wastes 3 KB. With millions of files, that waste can be gigabytes. Compression also becomes less effective on tiny files because the compression dictionary is per file.

Alternatives to Business-Key Splitting

If you need per-entity log access but want to avoid the resource problem, consider these alternatives:

• Structured logging + search: Write JSON logs to a central store and use a query engine (Elasticsearch, ClickHouse, BigQuery) to filter by key. This is the most scalable approach for high cardinality.
• Partition by time, not key: Split logs by hour/day, and include the key as a field. This keeps file counts low and still allows filtering.
• Hybrid: Write to a shared file, then have a separate process that extracts logs for specific keys into temporary files on demand (e.g., when a support ticket is created).
• Use a log management service: Managed services handle the scaling for you, but you still pay for the underlying resources, so the same trade-off applies.

Frequently Asked Questions

What is a business key in logging?

A business key is a field that identifies a business entity, such as a customer ID, order ID, or tenant ID. It’s used to group logs that belong to the same entity for easier retrieval and analysis.

How many log files is too many?

There’s no hard number, but a good rule of thumb is to keep the number of open files below 1,000 per process. If you exceed that, you’ll need to tune OS limits and watch for memory growth. Above 10,000, you’ll likely see performance degradation.

Can I split by multiple keys?

Yes, but the cardinality multiplies. If you split by customer and region, the number of streams is the product of distinct customers and regions. This usually makes the problem worse, so it’s better to use a single key or no split.

Is it better to split by time or by key?

For most systems, splitting by time is more resource-efficient because the number of time buckets is bounded (e.g., 24 per day). Splitting by key is only worth it if you have very few keys and need fast per-key access. Otherwise, use time-based partitioning and index the key.

What’s the difference between log splitting and log sharding?

They’re often used interchangeably, but sharding usually implies distributing logs across multiple servers or storage nodes, while splitting refers to separating logs into different files or streams on the same node. Both have the same resource implications.

How do I monitor for log splitting problems?

Watch for the number of open files, memory usage of the logging agent, and the time to rotate logs. If any of these grow linearly with log volume, you likely have a cardinality problem. Also, check for “too many open files” errors in the agent’s logs.

Related

• Question and Answer Sites: How They Work
• Roman Aqueducts: How Water Reached Higher Ground
• Pokémon Legends Z-A Roto-Glide: Unlock and Use It