Cloudflare Bot Management is designed to protect websites and web applications from malicious bot activities. It leverages machine learning, behavioral analysis, and fingerprinting techniques to accurately identify and mitigate harmful bots. Cloudflare offers customizable bot management rules, allowing businesses to tailor their protection strategies to specific needs. While Cloudflare’s bot protection rule for mitigating DDoS attacks can be crucial, this service might also block legitimate bots, such as server availability tracking bots, latency tracking bots, and any other bot configured by the website owner for legitimate usage.

To avoid False-Positive blocking of those legitimate bots, Red-Button recommends Clients who use Cloudflare Bots-Based WAF Rules take the following steps:

1. Deep-diving into Cloudflare WAF events to check for false activity: The first step is to check whether Cloudflare has blocked requests from your legitimate bots.
2. Allowing traffic from known legitimate bots to pass through the WAF: After seeing that Cloudflare did block your legitimate bots, a WAF bypassing rule should be configured for those IPs. This process can be configured in 2 methods:
   1. Adding the IPs/ASNs/IP ranges of the legitimate bots on Cloudflare’s IP Access Rules - in this method, we will let the legitimate bots bypass the WAF by adding their addresses to Cloudflare’s IP Access Rules:

Note that giving access to a whole ASN might let malicious bots associated with the AS bypass the WAF rules. In a condition where legitimate bots change their IPs, configuring specific IPs might not be effective.

1. Adding the IPs/ASNs/IP ranges of the legitimate bots to Cloudflare’s known bots list^[1] and configuring WAF rules in accordance - in this method, we will add the legitimate bots to Cloudflare’s verified bots list and configure a WAF rule that lets Cloudflare’s verified bots bypassing the WAF:

^[1] Cloudflare’s verified bots policy can be found here. There is no functional difference between “verified” and “known” bots.

 

Submitting a request to include your legitimate bots in Cloudflare’s Verified bots list may take a long time to answer as the process is up to Cloudflare.

3.  Determine whether the newly implemented configuration lets the legitimate bots access the website: After configuring one of the methods mentioned above, we now have to determine whether the configured rules allowed your legitimate bots to access the website without getting blocked. In addition, we would like to assure you that the implementation of those rules caused no malicious activity. To do so, we will use the following filters on  Cloudflare’s security event tracker, and then go deep into them :

 

                    method A - filtering events based on  the WAF rule ID                                        method B  - filtering events based on  the Access rules service