Monitoring Dashboard Data Integrity: What to Check

Your monitoring dashboard has lost old data integrity because of time-series database retention policies, downsampling aggregations, or schema migrations that altered or dropped historical metrics. To diagnose it, check your data source’s retention and rollup settings, verify the dashboard’s time range and query filters, and inspect the underlying storage for gaps or corrupted series.

Formula

The calculator estimates how many data points a monitoring dashboard can display for a given retention period and downsampling interval. The formula is:

Max data points = (Retention days × 24 hours/day × 60 minutes/hour) ÷ Downsample interval (minutes)

Variables:

• Retention (t) – how far back the data source keeps raw samples, in days.
• Downsample interval (d) – the time between aggregated samples (e.g., 1 minute, 5 minutes, 1 hour), in minutes.

The first output shows the maximum number of points the dashboard can plot if it queries the full retention range at the given interval. The second output shows the total raw samples stored (before any downsampling).

Worked Example

Using the defaults: retention = 30 days, downsample interval = 5 minutes.

1. Convert days to minutes: 30 × 24 × 60 = 43,200 minutes.
2. Divide by interval: 43,200 ÷ 5 = 8,640 points.
3. Raw samples: 30 × 24 × 60 = 43,200 samples.

The dashboard can show at most 8,640 points over 30 days with 5-minute aggregation, and the raw store holds 43,200 samples.

Why Old Data Disappears: Common Causes

Monitoring dashboards (Grafana, Datadog, Prometheus, InfluxDB, etc.) rarely delete old data by accident. More often, data integrity is lost through one of these mechanisms:

• Retention policies – time-series databases automatically drop raw samples older than a set period (e.g., 15 days for Prometheus). If your dashboard queries beyond that, older data simply isn’t there.
• Downsampling/rollups – systems like InfluxDB or VictoriaMetrics aggregate old raw samples into coarser intervals (e.g., 1-minute → 5-minute → 1-hour). The original fine-grained data is gone, so queries for high-resolution old data return fewer or different points.
• Schema changes – renaming a metric, changing tags/labels, or altering field types can make old series invisible to new queries. The data may still exist but under a different name.
• Backfill gaps – if the agent or collector was down, or the network failed, there are simply no samples for those time ranges. Dashboards show gaps, not errors.
• Time zone or UTC mismatches – a dashboard set to local time may display “missing” data that actually exists in UTC, especially around DST changes.

How to Check Your Data Source

Start with the storage layer, not the dashboard. Open your time-series database’s admin interface or CLI and run a raw query for the missing time range.

1. Check retention – look for the retention policy or TTL (time-to-live) setting. If it is shorter than your dashboard’s range, that explains the loss.
2. Inspect downsampling – many databases have continuous queries or aggregation tasks that replace raw data with rollups. Verify the interval and whether old raw data is deleted after rollup.
3. Query the exact metric – run a query for the metric name and time range you expect. If it returns nothing, try the same query with a broader time range or without label filters.
4. Check for schema changes – list all metric names or series in the database. If the old name is absent, look for a renamed version (e.g., http_requests → http_requests_total).
5. Verify agent uptime – if you use a local collector (Prometheus node_exporter, Telegraf, StatsD), check its logs and restart time. Gaps align with collector downtime.

What to Check in the Dashboard Itself

Sometimes the data is fine, but the dashboard query is wrong. Review these settings:

• Time range – if you select “Last 30 days” but the dashboard’s default is “Last 6 hours,” you’ll see only recent data. Check the time picker and any templated variables.
• Filters and regex – a label filter like env="prod" will exclude old series if they were tagged differently. Try removing filters to see if data appears.
• Downsampling in the query – many dashboards automatically downsample to keep queries fast. If the interval is too coarse (e.g., 1 day), old data appears as a single point per day.
• Cache or proxy – a caching layer (like Grafana’s caching or a reverse proxy) might serve stale or partial results. Clear the cache and re-run the query.
• Permissions – if you’re viewing a shared dashboard, your user role may restrict access to certain data sources or metrics. Compare with an admin account.

How to Prevent Future Data Loss

Once you’ve identified the cause, take these steps to protect historical data:

• Set retention to match your needs – if you need 90 days of raw data, configure the database to keep it that long. Be aware of storage costs.
• Use downsampling wisely – keep raw data for a short period (e.g., 7 days) and store aggregated data (e.g., 1-hour averages) for longer. This balances detail and cost.
• Back up the database – regular snapshots or exports to object storage (like S3) let you restore old data if needed.
• Monitor the monitor – set alerts for gaps in data collection, such as a metric that stops reporting for more than 5 minutes.
• Document schema changes – when renaming metrics or adding labels, update dashboards and alert rules in the same change request.

Common Misconceptions

“The data is still there, just hidden.” In many cases, it isn’t. Time-series databases are designed to delete old data automatically. Unless you have a backup, the data is gone.

“Downsampling preserves the same information.” It preserves an approximation. If you downsampled 1-minute data to 1-hour averages, you lose spikes and exact timestamps. That’s acceptable for long-term trends but not for incident analysis.

“A dashboard refresh will fix it.” Refreshing only re-runs the query; it cannot recover data that no longer exists in the database.

Frequently Asked Questions

Why does my Grafana dashboard show a gap for last week but data for yesterday?

This usually means your data source’s retention policy is shorter than the gap. For example, if retention is 7 days, last week’s data has been deleted. Check the database’s TTL or retention settings, and consider increasing retention or enabling downsampling to keep coarser data longer.

Can I recover deleted monitoring data?

Only if you have a backup or a separate archive. Some databases (like InfluxDB) have a “replay” feature if you stored raw data elsewhere. Without a backup, deleted time-series data is unrecoverable. Going forward, set up automated backups or export raw data to object storage.

Why does my Prometheus query return fewer points for older time ranges?

Prometheus automatically downsamples data when you query a long time range. The resolution is based on the range: the longer the range, the coarser the step. This is normal behavior, not data loss. To see more detail, query a shorter time range or use a recording rule to store higher-resolution aggregates.

What is the difference between retention and downsampling?

Retention is how long the database keeps any data. Downsampling is the process of aggregating multiple raw samples into one (e.g., averaging 5 minutes of data into one point). Downsampling reduces storage size but loses detail. You can have both: keep raw data for 30 days, then keep 1-hour aggregates for 2 years.

How do I check if my data source is actually receiving data?

Query the database directly for the metric and time range. If the query returns no results, check the collector’s status and logs. Also verify that the collector’s clock is synchronized (NTP); a clock skew can cause samples to be timestamped incorrectly and appear missing.

Why does my dashboard show data in UTC but not in local time?

This is a time zone conversion issue. Dashboards often default to UTC or browser time. If you’re in a time zone with DST, the conversion can shift data by an hour. Set the dashboard to UTC explicitly or use the correct time zone setting in the dashboard’s preferences.

Related

• What to Check Before Moving Workloads to an Iraq Data Center
• Windrose Grave Locations & Shipwreck Walkthrough
• FiveM b2372 GTA Crash Error: Fixes

in t | Retention (days) | days | 30 | 1 | 3650
in d | Downsample interval (min) | min | 5 | 1 | 1440
out points | Max data points | points | (t * 24 * 60) / d | 0
out raw | Raw samples | samples | t * 24 * 60 | 0