Troubleshoot Missing or Incomplete AI Agent Traces in Grafana

Missing or incomplete AI agent traces in Grafana usually point to a break somewhere between instrumentation, trace export, storage, and the Grafana query. Follow these checks in order to identify whether spans are not being created, transmitted, stored, or displayed; you’ll need access to the agent’s configuration and your telemetry pipeline.

What You’ll Need

• Access to the AI agent’s runtime configuration, logs, and deployment environment.
• Access to Grafana and the configured tracing data source, such as Tempo or another compatible backend.
• The service name, a recent trace or request timestamp, and, if available, a trace ID.
• Access to the OpenTelemetry Collector or other exporter and its logs, if your setup uses one.

Step-by-Step

1. Confirm that the request should produce a trace. Run one fresh agent request and note its exact time, service, environment, and outcome. Check whether tracing is enabled for that process and whether sampling is configured to retain the request. A request can succeed without producing a stored trace if sampling intentionally drops it. Test with a request known to be sampled, or temporarily use a safe, limited diagnostic setting in a non-production environment.
2. Check that instrumentation creates spans. Review the agent’s startup logs and instrumentation setup. Confirm the tracing SDK or supported instrumentation is initialized before the agent handles requests, and that the application creates a root span or equivalent. Look for initialization errors, missing instrumentation, or an agent framework integration that does not cover the operation you are investigating. If there are no spans at the source, changing Grafana queries will not fix the problem.
3. Verify trace context propagation. For calls between the agent, tools, model services, and other services, confirm that the current trace context is passed along. OpenTelemetry commonly propagates context using W3C Trace Context headers such as traceparent; verify that each relevant client and server preserves them. A broken boundary can create separate traces or leave a tool call without a parent span. Check proxies and custom HTTP clients that may remove or replace headers.
4. Inspect sampling and filtering. Check SDK sampling configuration, Collector processors, and backend policies. Head sampling may decide early whether to keep a trace; tail sampling may retain or discard it after observing spans. Filters can drop specific services, span names, attributes, or environments. Compare a missing request with a trace that appears, and inspect the effective configuration at each layer. Avoid disabling sampling broadly in production, since trace volume can rise quickly.
5. Follow the export path one hop at a time. Identify where the SDK sends spans: directly to the backend or to a Collector. Confirm the configured OTLP protocol and endpoint match the receiver, including whether the connection uses gRPC or HTTP and the expected path for HTTP. Check DNS, network access, TLS settings, authentication, and Collector receiver configuration. Inspect SDK and Collector logs for connection failures, rejected data, queue limits, or export timeouts.
6. Check Collector processing and delivery. If spans reach a Collector, review its pipeline to ensure the receiver, processors, and exporter are connected in the correct order. Look for memory limiting, batching, filtering, tail-sampling decisions, retries, and exporter errors. Check available Collector metrics for accepted, refused, sent, or failed spans. A queue that is full or an exporter that cannot reach storage can cause gaps even while the agent appears healthy.
7. Confirm that the backend received the trace. Use the tracing backend’s own search or trace-ID lookup, where available, before debugging Grafana panels. Verify the backend is healthy, receiving data, and retaining the relevant time range. Confirm that the trace has not expired and that its service or resource attributes match what you expect. If the backend cannot find it, focus on export, filtering, sampling, or ingestion rather than Grafana visualization.
8. Check Grafana’s data source and time range. In Grafana, open the relevant Explore view and select the correct tracing data source. Set an absolute time range around the recorded request time, allowing for clock differences, and search by trace ID if you have one. Check that the selected data source points to the intended environment and that its connection test succeeds. Verify the user has permission to query that data source.
9. Investigate incomplete traces by comparing spans. Open a trace that is present and inspect its span tree, timestamps, service names, and parent-child relationships. Find the first missing operation: was its span never created, detached into another trace, filtered, or lost during export? Check span status and attributes as well as the duration. Ensure the application ends spans when work completes, including error and cancellation paths; long-running or unclosed spans may appear unfinished or have misleading timing.
10. Validate changes with a controlled test. Make one configuration change at a time, send a new request, and search using its timestamp or trace ID. Confirm that the root span and expected child spans appear, and check Collector and backend counters for the same test. Record the final sampling, exporter, and resource-attribute settings so that a later deployment does not silently undo the fix.

Common Mistakes

• Searching the wrong data source, environment, service name, or time range.
• Assuming every request is recorded despite intentional sampling or filtering.
• Changing Grafana panels before confirming that the backend received spans.
• Using an endpoint or protocol that does not match the configured OTLP receiver.
• Expecting tool or model calls to join the parent trace without context propagation.
• Comparing clocks from machines with noticeable time differences; use a wider time window while diagnosing.

Tips

• Keep a single test request’s timestamp, trace ID, and service name together as you follow it through each layer.
• Check resource attributes such as service name and deployment environment for consistency across agent instances.
• Use a small, controlled test before changing sampling or Collector settings in production.
• When traces are partial, locate the first missing span in the call path; later spans may be healthy.
• Protect trace data. Span attributes can contain prompts, tool inputs, or other sensitive content; avoid collecting or sharing it unnecessarily.

Troubleshooting

No traces appear anywhere: Confirm instrumentation is active, sampling retains the test request, and the exporter is pointed at a reachable receiver. Check SDK startup logs, Collector receiver and exporter errors, and backend ingestion status.

The backend finds traces but Grafana does not: Select the correct tracing data source, widen the time range, search with the exact trace ID, and check data-source access and connection settings. Make sure the backend queried by Grafana is the same one receiving the spans.

Only some agent steps appear: Check context propagation at service boundaries, then inspect processors and sampling rules for filters that match particular services or span attributes. Confirm that each operation is instrumented and that spans are ended on success, failure, and cancellation.

Traces arrive late or with gaps: Look for Collector queues, retries, backpressure, exporter timeouts, and backend ingestion delays. Compare timestamps across components and check whether resource limits or retention settings affect the data. If the cause is unclear, capture a short test’s SDK and Collector logs and escalate with the trace ID, configuration summary, and timestamps—excluding sensitive span content.

Frequently Asked Questions

Why does an agent request succeed but have no trace?
A successful request does not guarantee a trace. Instrumentation may be disabled, sampling may drop it, or export and ingestion may fail independently of the request.

Can Grafana create missing spans?
No. Grafana queries and displays stored telemetry; spans must be created by instrumentation and delivered to the tracing backend first.

Why are tool calls shown as separate traces?
The trace context may not be passed to the tool client or preserved by an intermediary. Check propagation headers and instrumentation on both sides of the call.

How can I tell whether sampling is the cause?
Compare the effective SDK, Collector, and backend sampling policies with a controlled test. Check logs or metrics for sampling decisions where available; do not assume a missing trace alone proves sampling.

What information should I collect before asking for help?
Provide a trace ID if available, request timestamp and time zone, service and environment, relevant exporter or Collector errors, and the affected configuration. Remove prompts, credentials, and other sensitive span data.

Related

• Fix Corrupted Game DB on Conan Exiles Server
• Unicode Cursive Text in Passwords & Usernames: Safe?
• Securing AI Agents & Credentials in Production