How to See Which Bots Visit Your Angular Site

This guide shows you how to identify which bots and AI crawlers (like Googlebot, Bingbot, GPTBot, and others) are hitting your Angular site. You’ll learn to log incoming requests, filter for known bot user agents, and view a clear report—either in your browser console or a simple backend endpoint. You’ll need a basic Angular project (any version) and, optionally, a backend server if you want to store logs beyond the current session.

What You’ll Need

• An Angular application (Angular 2+; the steps work with any version).
• Node.js and npm installed (for running Angular CLI commands).
• Basic familiarity with Angular services and HTTP interceptors.
• Optional: A backend server (like Express) if you want to persist logs.

Step-by-Step

1. Create a new Angular service for logging. Run ng generate service bot-logger in your terminal. This creates bot-logger.service.ts.
2. Inject the service into your app. Open app.module.ts (or app.config.ts for standalone) and add BotLoggerService to the providers array. This makes it available app-wide.
3. Implement the logging logic. In bot-logger.service.ts, add a method like logBotRequest() that reads the user agent from the request headers. In the browser, you can access it via navigator.userAgent.
4. Create an HTTP interceptor. Run ng generate interceptor bot-detector. This interceptor will catch every outgoing HTTP request from your Angular app.
5. Intercept requests. In the interceptor, implement the intercept method. It receives the request and next handler. Before passing the request, check if the user agent matches a known bot pattern.
6. Define a bot user-agent list. Create a constant array of regex patterns for common bots: /googlebot/i, /bingbot/i, /gptbot/i, /claudebot/i, /applebot/i, /duckduckbot/i, etc. You can find a comprehensive list online.
7. Log the bot. If the user agent matches, call your logger service to record the bot name, timestamp, and requested URL. For a simple in-browser log, use console.log or store in an array.
8. Send logs to a backend (optional). If you want persistent logs, modify the logger to POST the data to your own server endpoint. The interceptor can send this log request, but be careful to avoid recursive logging (exclude your logging endpoint from interception).
9. Test locally. Run ng serve and open your app. Use browser dev tools to simulate a bot user agent (e.g., in Chrome DevTools, click the three-dot menu > More tools > Network conditions, then override the user agent).
10. View the logs. Open the browser console (F12) to see the logged bot visits. If you sent them to a backend, query your database or log file.

Common Mistakes

• Only checking the browser user agent. Angular runs in the browser, so you only see the user agent of the client making the request. Server-side rendering (SSR) or a backend is needed to see the original bot’s user agent if your site is pre-rendered.
• Blocking all bots by mistake. Some bots (like Googlebot) are essential for SEO. Make sure you only log, not block, unless you have a specific reason.
• Forgetting to exclude your own logging endpoint. If you send logs to your backend, the interceptor will also log those requests, causing an infinite loop. Always exclude the logging URL.
• Using outdated bot lists. New AI crawlers appear frequently. Keep your regex list updated.
• Assuming all bots have a user agent. Some malicious bots may spoof or omit user agents. Log any request with a missing or empty user agent as a potential bot.

Tips

• Use a dedicated library. Consider using isbot or ua-parser-js to detect bots more accurately. These libraries are well-maintained and cover many edge cases.
• Store logs in a structured format. JSON is ideal for later analysis. Include fields like timestamp, userAgent, url, and ip (if available).
• Add a robots.txt file. While this doesn’t log bots, it tells them which paths to crawl. You can create a robots.txt in your src folder and add it to your build.
• Use Angular Universal (SSR). With SSR, your server can access the original request headers, giving you accurate bot detection even for pre-rendered pages.
• Monitor logs regularly. Set up a simple dashboard or alert if you notice an unusual spike in bot traffic.

Troubleshooting

• No bots are logged. Check that your interceptor is properly registered in the providers array. Also, verify your regex patterns against real bot user agents (e.g., Googlebot’s user agent contains “Googlebot”).
• Logs appear for human visitors. Your bot detection might be too broad. For example, a pattern like /bot/i will match “robot” in a human user agent. Use specific patterns.
• Infinite loop when sending logs to backend. Exclude the logging endpoint in your interceptor by checking the request URL. For example, if (req.url.includes('/api/log-bot')) return next.handle(req);
• User agent is empty in some browsers. Some privacy-focused browsers may obscure the user agent. In that case, you might miss some bots. Use other signals like IP or behavior patterns, but that’s more complex.
• Bots are not hitting your Angular app. If you have a server-side redirect or a CDN, bots might be cached at that level. Check your server logs as well.

Frequently Asked Questions

Can I see bots that visit my Angular site without a backend?

Yes, you can log them in the browser console or store them in memory, but those logs disappear when the page reloads. For persistent logs, you need a backend or a third-party analytics service.

Does Angular itself have built-in bot detection?

No, Angular does not provide bot detection out of the box. You need to implement it using interceptors or services.

Will logging bots slow down my site?

Minimal. Logging a few strings per request is negligible. However, if you send logs to a backend on every request, it could add latency. Batch logs or send asynchronously.

Can I block AI crawlers like GPTBot?

Yes, you can block them by adding rules to your robots.txt or by returning a 403 status from your server. However, be aware that some AI crawlers may ignore robots.txt.

How do I know if a bot is actually a search engine bot?

You can perform a reverse DNS lookup on the IP address to verify the bot’s identity. For example, Googlebot IPs resolve to *.googlebot.com. This is more reliable than just the user agent.

Do I need to log bots if I use a CDN?

If your CDN caches your site, bots may be served from the CDN edge, and your Angular app may not see the request. Check your CDN logs for bot traffic.

Related

• Vast 4.0 Root Level Error & Ad: Why No Shared Response?
• Recover Galaxy S9 Data Files: Full Guide
• FileZilla Upload Files to Server: Step-by-Step