Configure a WAF to Stop Malicious Automated Traffic

This guide walks you through configuring a Web Application Firewall (WAF) to filter out malicious automated traffic—such as credential stuffing, content scraping, and DDoS floods—while letting legitimate users through. You’ll learn how to set up rate limiting, bot detection, and challenge rules, and how to test and refine them. You’ll need administrative access to your WAF (cloud-based or on-premises) and a basic understanding of your application’s traffic patterns.

What You’ll Need

• Administrative login to your WAF console or management interface.
• Access to your web server logs or analytics to identify normal traffic baselines.
• A list of known good IPs or user agents (if you have any).
• Ability to test from both a normal browser and a scripted client (like cURL or a bot).

Step-by-Step

1. Identify your traffic baseline. Before making changes, review your logs for the last 30 days. Note the average requests per second (RPS) per IP, typical user agents, and peak hours. This gives you a reference to avoid blocking legitimate users.
2. Enable the WAF in log-only mode. Most WAFs allow you to deploy rules without actively blocking. Turn on logging for all rules and run for 24–48 hours to see what would have been blocked. This prevents accidental outages.
3. Configure rate limiting. Create a rate limiting rule that caps requests per IP. Start with a generous limit—e.g., 300 requests per minute for a typical web app—and set a burst allowance of 100. Use the log data to adjust: if your heaviest legitimate user hits 200/min, set the limit to 250/min.
4. Set up bot detection. Enable the WAF’s bot management features. Mark known good bots (Googlebot, Bingbot) as allowed. For unknown bots, configure a challenge action (like a JavaScript or CAPTCHA challenge) rather than an outright block, so human users with unusual setups aren’t lost.
5. Create rules for suspicious patterns. Add rules to block or challenge traffic with common automated signatures: user agents containing “python-requests”, “curl”, or “scrapy”; requests with no referrer; or high request rates to login or API endpoints. Use the “challenge” action for borderline cases.
6. Implement IP reputation filtering. If your WAF has a threat intelligence feed, enable it to block or challenge traffic from known malicious IPs. This is often a one-click toggle, but you can also add custom lists of IPs you’ve identified as abusive.
7. Set up geo-blocking (optional). If your business only serves certain countries, block or challenge traffic from other regions. Be cautious—this can block legitimate travelers or VPN users.
8. Enable active mode gradually. After 48 hours of log-only, switch the rules to active mode for a small subset of traffic (e.g., 10% of requests) and monitor for false positives. Then increase to 50%, then 100%.
9. Test thoroughly. Use a normal browser to confirm access works. Use a scripted client (like cURL) to confirm it gets blocked or challenged. Also test edge cases: a user behind a corporate proxy (shared IP) or on a slow connection.
10. Monitor and tune. After going live, review blocked requests daily for the first week. Whitelist any legitimate IPs that got blocked, and adjust rate limits if you see false positives. Set up alerts for sudden spikes in blocked traffic.

Common Mistakes

• Setting rate limits too low. Users behind a NAT or corporate proxy share an IP, so a limit like 10 requests/minute will block entire offices. Always use your baseline data.
• Blocking all non-browser user agents. APIs and mobile apps often have custom user agents. If you block them, you break your own services. Use challenge instead of block for unknown agents.
• Ignoring false positives. If you don’t monitor, you’ll silently block legitimate customers. Always keep logging enabled and review.
• Deploying all rules at once. Changing everything simultaneously makes it hard to pinpoint which rule caused an issue. Roll out incrementally.
• Forgetting to whitelist your own IPs. You’ll lock yourself out of the admin panel if you’re not careful. Add your office IPs to a whitelist first.

Tips

• Use the “challenge” action more often than “block”. Challenges stop bots but let humans through with minimal friction.
• Combine rate limiting with bot detection—bots can often bypass simple rate limits by rotating IPs.
• Set up separate rules for your API endpoints, as they often have different traffic patterns than web pages.
• Use the WAF’s built-in dashboard to visualize blocked traffic and identify new attack patterns.
• Schedule regular reviews (monthly) to update your rules as your traffic evolves.

Troubleshooting

• Legitimate users are being blocked. Check the WAF logs for the blocked IP. If it’s a shared IP, increase the rate limit or add the IP to a whitelist for specific paths. Also review your bot detection rules—maybe a legitimate user agent is being misclassified.
• Bots are still getting through. Verify that your rules are in active mode, not just log-only. Check if the bots are using rotating IPs—if so, rely more on behavior-based detection (e.g., rate of requests to login) rather than IP-based rules.
• Your own API calls fail. Add your API client’s user agent to the allowed list, or create a separate rule that bypasses bot detection for authenticated API requests.
• Traffic spikes after enabling WAF. This could be a DDoS attack, but also could be a misconfigured rule causing retries. Check if the spike is from a single IP or distributed—if distributed, it’s likely an attack.
• WAF is slowing down your site. Some actions (like JavaScript challenges) add latency. If performance degrades, reduce the use of challenges on static resources, or upgrade your WAF plan if you’re on a cloud provider.

Frequently Asked Questions

What is the difference between a WAF and a bot manager?

A WAF filters HTTP traffic based on rules (like rate limits and signatures). A bot manager uses more advanced techniques like behavioral analysis and device fingerprinting to distinguish humans from bots. Many modern WAFs include bot management features, but they can be separate products.

How do I know what rate limit to set?

Look at your analytics or server logs to find the maximum requests per minute for your top 1% of IPs. Set your limit slightly above that (e.g., 1.5x). If you don’t have data, start with 100 requests per minute and adjust based on false positives.

Will a WAF block all automated traffic?

No. It can block or challenge most malicious bots, but sophisticated bots that mimic human behavior may evade detection. That’s why you need layered defenses like strong authentication and monitoring.

Can I use a WAF to stop DDoS attacks?

A WAF can mitigate application-layer DDoS attacks (like HTTP floods) by rate limiting and challenging suspicious traffic. For network-layer DDoS (like SYN floods), you need a separate DDoS protection service.

How long should I run in log-only mode?

At least 24–48 hours, ideally over a weekend or a full business cycle, to capture both peak and off-peak traffic. This gives you a reliable baseline to set limits.

What should I do if I accidentally block a legitimate user?

Whitelist their IP address immediately, and then review the rule that caused the block to see if it needs adjustment. Also consider adding a custom error page that explains how to contact support if they’re blocked.

Related

• Haze Seas Swords Tier List Guide
• Google Wallet: Receipts, Passes, Cards & Device Support
• Jio World Record: 5G Speed Test in India