Many public or commercialized programs, including search engine crawlers, have fixed or default User-Agent header features and have specific purposes. Bot management policies include most public bot type features, and you can directly manage bot tools that meet these features, which can help you:
1) Allow search engine crawlers to access and avoid being blocked wrongly;
2) Identify specific-purpose commercialized tools and limit their access.
EdgeOne will regularly update the features of automated tools to ensure that your management strategy continues to cover control scenarios.
Usage Scenarios
By default, the Basic bot management strategy is in a disabled state. When you have the following scenario demands, you can enable and adjust the bot basic management protection strategy as needed:
Control requests from IDC (data center)
Most of the access to To C applications comes from mobile networks, broadband providers, or educational networks, and normal requests do not come from data centers (IDC). Therefore, requests from cloud providers or data centers are mostly from proxies or crawlers. You can choose to control requests from data centers (IDC) and intercept or perform JavaScript challenges to mitigate the risk of malicious access.
Control valid bot requests with search engine features
Search engine crawlers are currently one of the few valid bot types. In order for sites to distinguish valid crawlers from search engines, most search engine providers provide the IP segment and UA features used by their crawler engines. EdgeOne's search engine feature rules include search engine public IP features, User-Agent header features, rDNS resolution features, and other matching methods. You can configure bot requests with search engine features to be released to avoid being intercepted by bot management policies.
Control requests from commercial or open-source tools
Commercial software or open-source tools often carry specific User-Agent features. EdgeOne classifies these automated tools based on their usage and regularly updates the corresponding User-Agent library. If you do not allow bot requests from these commercial or open-source tools, you can intercept them.
Adjust Basic Management Protection Strategy
For example, your current site shop.example.com is an e-commerce website. In order to prevent users from placing orders and snatching purchases through tools, you need to disable the automatic shopping cart bot. You can follow the steps below:
1. Log in to the EdgeOne console, click on the site list in the left menu bar, click on the site to be configured in the site list, and enter the site details page.
2. On the site details page, click Security Protection > Bot Management to enter the Bot Management details page.
3. In the Basic Bot Management Settings card, click Set to enter the configuration page.
4. Select UA feature rules and click on the detailed rules in the upper right corner.
5. In the detailed rules page, you can modify the action for a specified rule ID individually; if you need to configure in batches, you can also click Batch Settings, select the rule IDs to be configured in batches, choose the action, and apply.
In this scenario, you can modify the action for the automatic shopping cart robot to Block.
Yes
No
Was this page helpful?