keepcalculator

06 Numbers and Powers

Event Log Best Practices: What to Record

Card No. 06·80417 · Updated · Standard formula, shown below

Throughput (KB/s) 8.53 KB/s
Duration (minutes)
2.00 min
Result
Event duration (seconds)
120 sec
You entered
Data size (KB)
1024 KB
You entered
You enter
Event duration (seconds), Data size (KB)
You get
Throughput (KB/s), Duration (minutes)
Rounding
2 dp

An event log should record the who, what, when, where, and outcome of every significant action or state change in a system, with enough context to reconstruct the sequence of events and diagnose problems without requiring the original user to explain what happened. The core fields are a timestamp, the actor (user, service, or system), the action or event type, the target resource, the result (success/failure), and any details that make the event meaningful for security, debugging, or compliance. This article explains what to record, why each field matters, and how to choose the right level of detail for your system.

Formula

The calculator above computes throughput (data processed per second) from a duration and data size, which is a typical metric you might derive from event log timestamps. The formula is:

Throughput = Data size / Duration

Where:

  • Data size (KB) – the amount of data processed, logged, or transferred.
  • Duration (seconds) – the time span between the start and end events in the log.
  • Throughput (KB/s) – the average rate of processing, useful for spotting slow operations.

Worked Example

Using the default values from the calculator: an event took 120 seconds and processed 1024 KB of data.

  1. Throughput = 1024 KB / 120 s = 8.533… KB/s
  2. Rounded to two decimals: 8.53 KB/s
  3. Duration in minutes = 120 / 60 = 2.00 minutes

This matches the calculator output. The same logic applies to any pair of timestamps in an event log: subtract the start time from the end time to get duration, then divide the work done by that duration.

Why Event Logs Matter

Event logs serve three main purposes: troubleshooting (finding why something failed), security (detecting unauthorized access or anomalies), and compliance (proving what happened for audits). Without a well-structured log, you are blind to what your system actually did. A good log answers questions like: Who logged in at 3 AM? Why did the database crash? Did the backup job finish last night? Each recorded event is a piece of evidence that helps you reconstruct the past.

Core Fields Every Event Should Have

While specific needs vary, most event logs should include these essential fields:

  • Timestamp – with timezone and UTC offset. Use UTC internally to avoid ambiguity across servers.
  • Event ID or type – a stable identifier (e.g., “LOGIN_SUCCESS”, “FILE_DELETE”) so you can filter and group events.
  • Actor – the user ID, service account, or system component that initiated the action.
  • Action – the verb: create, read, update, delete, execute, etc.
  • Target – the resource affected (file path, database record, IP address, etc.).
  • Result – success, failure, or partial success, plus an error code or message.
  • Source IP and user agent – for network-based events, helps identify origin.
  • Correlation ID – a unique identifier that links related events across services (useful in microservices).
  • Additional context – any data specific to the event, such as request parameters, file sizes, or previous state.

How to Choose What to Record

Start with the questions you need to answer. If you run an e-commerce site, you might need to know every order step: cart added, payment attempted, payment succeeded, order shipped. If you manage a network, you might need login/logout events, firewall drops, and configuration changes. A practical approach is to log anything that could be relevant to a security incident or a user-facing error. For each event type, ask: “If this happens and we don’t log it, will we regret it?” If yes, log it.

Also consider the principle of least privilege for logging: log the minimum necessary to meet your operational and compliance needs. Over-logging can be as harmful as under-logging because it floods your storage and makes it hard to find the important events.

Common Problems and Misconceptions

One common mistake is logging too little context. For example, logging “User failed to log in” without the username makes the log useless for detecting brute-force attacks. Another is using local time without timezone – this breaks when servers are in different time zones or during daylight saving changes. Also, many people forget to log the outcome of a process, not just the start. A log that says “Backup started” but never says whether it completed is only half useful.

Another misconception is that logs are immutable. In practice, logs can be altered by an attacker with root access. For security-critical logs, consider write-once media or a separate log server with strict access controls. Also, logs are not a substitute for monitoring – they are the raw data that monitoring systems aggregate and alert on.

Structured vs. Unstructured Logs

Structured logs (JSON, key-value pairs) are easier to search, parse, and analyze programmatically. Unstructured logs (free text) are easier to read but harder to query. Modern best practice is to use structured logging, even if you also include a human-readable message. For example:

{"timestamp":"2025-01-01T12:00:00Z","event":"LOGIN","user":"alice","ip":"192.168.1.1","result":"success"}

This is far more useful than a plain text line like “Alice logged in successfully”.

Retention and Storage Costs

Logs can consume enormous storage. A busy server can generate gigabytes per day. You need a retention policy that balances cost with compliance requirements. Typically, you keep raw logs for a short period (e.g., 30 days) for debugging, and aggregated summaries for longer (e.g., 1 year) for trend analysis. Use log rotation to compress and archive old logs. Consider using a log management service that can handle large volumes and provide search capabilities.

Security and Privacy Considerations

Logs often contain sensitive data: passwords (never log those), personal information, session tokens, or credit card numbers. You must redact or encrypt such fields before storing logs. Also, logs themselves are a target for attackers – if they can delete or modify logs, they can cover their tracks. Protect your log files with proper file permissions, and consider using a separate logging server that is not accessible from the application servers.

Frequently Asked Questions

Should I log every single action a user takes?

Not necessarily. Log actions that are security-relevant (logins, permission changes, data deletion) or that could cause user-visible errors. For high-volume actions like page views, you might use sampling or aggregate counts instead of individual logs.

What is a correlation ID and why do I need it?

A correlation ID is a unique string generated at the start of a request and passed to all services involved. It lets you trace a single user action across multiple logs, which is essential in microservices architectures where a single request may hit many services.

How long should I keep event logs?

It depends on your industry and regulations. For general troubleshooting, 30-90 days is common. For compliance, you may need to keep logs for years. Check applicable laws (like GDPR or HIPAA) and your company’s data retention policy.

What’s the difference between a log and a metric?

A log is a discrete event with details; a metric is a numeric measurement over time (e.g., CPU usage, request count). Logs answer “what happened?” and metrics answer “how much?” Both are useful; often you use metrics to detect a problem and logs to diagnose it.

Can I use my application’s console output as an event log?

Console output is often unstructured and may lack critical fields like timestamps or user IDs. It’s better to use a proper logging library that outputs structured data to a file or log aggregator. Console output is fine for development, but not for production.

What should I do if I find a log entry that seems wrong?

Investigate before deleting it. Logs are evidence; altering or deleting them can be worse than keeping a bad entry. If the log is incorrect, note the issue and fix the logging code, but preserve the original entry as-is.

Related

Formula used
in t | Event duration (seconds) | sec | 120 | 0 | 86400
in d | Data size (KB) | KB | 1024 | 0 | 1048576
out rate | Throughput (KB/s) | KB/s | d / t | 2
out mins | Duration (minutes) | min | t / 60 | 2

Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →