Connect AI Coding Agents to Observability Data Securely

This guide shows you how to give AI coding agents (like Copilot, Cursor, or custom LLM workflows) read-only access to your application’s observability data—logs, metrics, and traces—without ever exposing production credentials. You’ll set up a secure proxy that authenticates on behalf of the agent, scopes permissions to read-only, and rotates secrets automatically. By the end, your agent can answer questions like “why did error rates spike at 3 PM?” using real production data, while your database passwords and API keys stay hidden.

What You’ll Need

An observability platform (e.g., Datadog, Grafana, Prometheus, New Relic, or a self-hosted stack) with an API that supports scoped API tokens.
Access to a small server or serverless function (e.g., AWS Lambda, Cloudflare Worker, or a simple VPS) to run the proxy.
Basic familiarity with environment variables, HTTP requests, and JSON.
Your AI coding agent must support custom API endpoints or MCP (Model Context Protocol) tools—most modern agents do.

Step-by-Step

Create a read-only service account in your observability platform. Most platforms let you create an API token with scopes. Choose the most restrictive scope that still allows reading the data you need. For example, in Grafana, create a service account with only the “Viewer” role; in Datadog, create an API key with the “read” permission only. Do not use your personal admin token.
Set up a proxy server. Create a new directory on your server or serverless function. We’ll use a simple Node.js Express server as an example, but any language works. Initialize a project and install dependencies: npm init -y && npm install express dotenv.
Store the observability token as an environment variable. Create a .env file in the proxy directory (and make sure it’s gitignored). Add: OBS_TOKEN=your_read_only_token_here. Also add OBS_BASE_URL=https://api.yourobservability.com. Never hardcode these.

Write the proxy logic. Create an index.js file. The proxy will accept requests from the AI agent, add the observability token as a header, forward the request to the observability API, and return the response. Crucially, the proxy must not expose the token to the agent. Here’s a minimal example:

const express = require('express');
const app = express();
require('dotenv').config();

app.use(express.json());

app.all('/proxy/*', async (req, res) => {
  const target = req.url.replace('/proxy/', '');
  const url = `${process.env.OBS_BASE_URL}/${target}`;
  const headers = {
    'Authorization': `Bearer ${process.env.OBS_TOKEN}`,
    'Accept': 'application/json'
  };
  try {
    const upstream = await fetch(url, {
      method: req.method,
      headers,
      body: req.method === 'GET' ? undefined : JSON.stringify(req.body)
    });
    const data = await upstream.json();
    res.status(upstream.status).json(data);
  } catch (err) {
    res.status(500).json({ error: 'Proxy error' });
  }
});

app.listen(3000, () => console.log('Proxy on :3000'));

Add request validation and rate limiting. To prevent abuse, only allow GET requests (read-only) and limit the paths the agent can access. For example, block any path that contains “delete”, “update”, or “write”. Also add a simple rate limiter like express-rate-limit to avoid runaway requests.
Deploy the proxy. If using a VPS, run node index.js behind a reverse proxy like Nginx with HTTPS. If using a serverless function, adapt the code to the platform’s handler. Ensure the proxy is only accessible over HTTPS.
Expose the proxy to your AI agent. Most coding agents support custom tools or MCP servers. For MCP, create a simple MCP server that calls your proxy endpoints. For example, in Cursor, you can add a custom API endpoint under Settings > Tools. Point the agent to https://your-proxy-domain.com/proxy/ with no authentication headers—the proxy handles auth.
Test with a safe query. Ask the agent to fetch a list of recent errors from the proxy. For example, in your agent’s prompt: “Call the proxy endpoint /proxy/api/v1/errors?limit=5”. Verify the agent gets the data and that the token is not visible in the agent’s logs.
Set up token rotation. Manually rotate the token in the observability platform every 30-90 days, or use a secrets manager like AWS Secrets Manager to auto-rotate. Update the proxy’s environment variable without restarting if possible (e.g., using a secrets file that the proxy reads on each request).
Monitor the proxy. Add logging to the proxy (without logging the token) so you can see which endpoints the agent calls. Set up alerts for unusual traffic patterns.

Common Mistakes

Using a personal admin token. This gives the agent full access to all data and operations. Always create a dedicated read-only token.
Exposing the token in the agent’s environment. If you set the token as an environment variable on your laptop, the agent might pick it up. Keep it only on the proxy server.
Not restricting HTTP methods. If your proxy forwards any method, the agent could accidentally (or maliciously) delete data. Force GET only.
Logging the Authorization header. Many web servers log request headers by default. Disable that or sanitize logs.
Hardcoding the token in the proxy code. If your repository is public, the token leaks. Always use environment variables or a secrets manager.

Tips

Use a short token validity period if your platform supports it, and rotate frequently.
Add a request size limit to prevent the agent from pulling huge logs that could slow down your system.
If your observability API supports query parameters, restrict the proxy to only allow read-only query types (e.g., no “export” endpoints).
Consider using a dedicated subdomain for the proxy to keep it separate from your main app.
For MCP, use the official MCP SDK to build a proper server with tool definitions—this gives the agent structured access.

Troubleshooting

The agent gets a 401 Unauthorized. Check that the token is correctly set in the proxy’s environment and that it hasn’t expired. Also verify that the token has read permission on the specific API endpoint.
The agent gets a 403 Forbidden. Your token might be scoped to a specific project or team. Ensure the token has access to the data you’re trying to query.
The agent gets a CORS error. If the agent runs in a browser (e.g., a web-based IDE), add CORS headers to the proxy: Access-Control-Allow-Origin: * and handle preflight OPTIONS requests.
The proxy returns a 500 error. Check the proxy logs. The most common cause is a malformed URL or the observability API returning a non-JSON response. Add error handling for non-JSON responses.
The agent sees the token in its debug logs. This can happen if the agent logs the full request. Ensure your proxy does not echo the Authorization header in any response body or error message.
The proxy is slow. Add caching for common queries, or increase the timeout on the upstream request. Also consider using a serverless function that scales automatically.

Frequently Asked Questions

Can I use the same proxy for multiple AI agents?

Yes. The proxy is agent-agnostic. Just point each agent to the same proxy URL. You can add per-agent API keys later if you need to track usage.

What if my observability platform doesn’t support scoped tokens?

Use a full token but restrict the proxy to only allow GET requests and whitelist specific endpoints. Also, rotate the token often. If the platform supports IP whitelisting, restrict the token to your proxy’s IP.

How do I handle authentication for the proxy itself?

If the proxy is publicly accessible, add a simple shared secret that the agent must include in a header (e.g., X-Proxy-Key). The proxy checks this before forwarding. This prevents random internet users from hitting your proxy.

Can I use this with a self-hosted observability stack like Prometheus?

Yes. The same pattern works. Instead of an API token, you might use basic auth or a bearer token. The proxy just adds the necessary authentication header.

What if the agent needs to write data (e.g., create alerts)?

That’s risky. If you must allow write, create a separate proxy endpoint that only allows specific write operations with strict validation. But generally, keep the agent read-only to avoid accidental changes.

How do I know the agent is not leaking data?

Review the proxy logs regularly. Also, set up alerts if the proxy receives requests outside of working hours or from unexpected IPs. You can also add a data redaction layer in the proxy to strip sensitive fields (like user emails) before returning data to the agent.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →