With the rapid development of generative AI and large-scale model technology, AI crawler traffic for model training and information search is exponentially growing on the Internet. The proportion of such AI crawlers in overall internet traffic continues to rise, and some users expect to control this part of traffic. Meanwhile, in marketing scenarios, there is also a need to leverage their content indexing and spread capacity to reach more users through AI applications and expand exposure, requiring permission for related AI crawlers to access resources. The AI crawler control feature identifies traffic characteristics of mainstream AI crawlers, thereby achieving specific control over crawler requests.
Directions
1. Log in to the Tencent Cloud EdgeOne console, enter Service Overview in the left menu bar, and click the site to be configured under Website Security Acceleration.
2. Click Security > WebSecurity . By default, it is a site-level security policy. Click the Domain-level security policy tab and then click the target domain name such as shop.example.com , to enter the configuration page for the security policy of the target domain name.
3. Locate the Bot management card and enter the Basic Bot management > AI crawler Control page.
4. Click edit to configure the action for AI crawlers. The AI crawler control feature supports Observe, Block, Allow, JavaScript challenge, and Managed challenge. According to compatibility configuration, select the appropriate option. For details, see action.
5. Click Save to complete the configuration.
Note:
1. AI crawler control performs traffic analysis based on the User-Agent field of the request. If needed, subscribe to the Bot management feature to analyze and control requests based on behavioral analysis, IP portrait, request rate, and other intelligent analysis functions.
2. If normal traffic is misintercepted, configure the exception rule to restore the normal traffic.
Reference
1. When AI crawler control is enabled, EdgeOne will identify and control requests from the following crawlers.
