Tencent Cloud

Recent Pages

Rate Limiting

Last updated: 2023-12-18 15:31:41
Overview
In site operation, problems such as malicious resource occupation, business abuse, and brute force cracking often occur. If these problems are ignored, they will lead to a decline in service quality, generate high-cost bills, and may even cause sensitive data leakage. To effectively manage these risks, client access frequency is an important indicator. Malicious clients usually access at a higher frequency to quickly achieve the purpose of cracking login, occupying resources, and crawling content. Using appropriate threshold limits for client access frequency can effectively distinguish between normal clients and malicious clients, thereby mitigating the risks of resource occupation and abuse.
Note：
When managing and combating crawlers, the effect of using only rate limiting strategy is limited. Please combine Bot management function to formulate a complete crawler management strategy.
Typical Scenarios and Usage
Rate limiting is commonly used to distinguish between normal client access and malicious access. By selecting appropriate statistical methods, limit thresholds, and disposal methods, rate limiting can help you mitigate security risks. Rate limiting configuration is divided into the following types:
Accurate matching rules: User-defined access frequency control strategy. Supports multiple condition combinations to match requests, limit the request rate of each request source, and is suitable for most scenarios to distinguish between normal user access and malicious high-frequency access.
Managed custom policies: Policies customized by Tencent security experts, which do not support console adjustment of policies. For details, please refer to Managed Custom Rules.﻿
Accurate Matching Rules
Example Scenario 1: Limit the access frequency of the login API interface to mitigate credential stuffing and brute force cracking attacks
In the face of credential stuffing and brute force cracking attacks, attackers often frequently use access to the login API interface to try to obtain or crack information. By limiting the request frequency of the login interface, we can significantly mitigate the attacker's cracking attempts, effectively defend against such attacks, and protect sensitive information from being leaked.
For example: The domain name www.example.com provides an external interface /api/UpdateConfig, the allowed access call frequency is 100 times/minute, and when the frequency limit is exceeded, the IP will be blocked for 10 minutes. The operation steps are as follows:
1. Log in to the EdgeOne console and click Site List in the left sidebar. In the site list, click the target site.
2. In the site details page, click Security Protection > Web Protection to enter the Web Protection details page, and select the domain name to be protected in the left protection domain list.
﻿
3. Find the rate limiting card and click Settings. Enter the rate limiting configuration page and click Add Rule in the Accurate Rate Limiting Rules.
﻿
4. In the pop-up rule page, configure according to the following steps:
4.1. Fill in the rule name and select the custom protection object for the matching object.
4.2. In the matching condition list option, configure the matching condition of the rule. In this scenario, select the request path equal to /api/UpdateConfig.
4.3. Configure the triggering method of this rule. In this scenario, configure the counting period of 60 seconds, and trigger when the count exceeds 100 times. The statistics method is triggered when a single client IP requests to the EdgeOne node, and after triggering, the triggering state is maintained for 10 minutes.
4.4. Select Intercept for the execution action. The complete rule configuration is as follows:
﻿
5. After clicking OK, the rule will be deployed and take effect.
Example Scenario 2: Limit the request rate causing 404 status code to mitigate random resource scanning
When malicious clients randomly scan site image resources and try to crawl content, they often cause the origin server to respond with a 404 error due to non-existent access paths. By limiting the request rate that causes the origin server's 404 status code, EdgeOne can prevent malicious attackers from scanning and requesting static resources on a large scale, thereby reducing the origin server's error response, alleviating server pressure, and improving the security and stability of static resource sites. For example: For the domain name www.example.com's image static resources .jpg .jpeg .webp .png .svg, when the resource does not exist and responds with a 404, if the access exceeds 200 times within 10 seconds, the corresponding client IP request will be directly blocked for 60 seconds. The operation steps are as follows:
1. Log in to the EdgeOne console and click Site List in the left sidebar. In the site list, click the target site.
2. In the site details page, click Security Protection > Web Protection to enter the Web Protection details page, and select the domain name to be protected in the left protection domain list.
﻿
3. Find the rate limiting card and click Settings. Enter the rate limiting configuration page and click Add Rule in the Accurate Rate Limiting Rules.
﻿
4. In the pop-up rule page, configure according to the following steps:
4.1. Fill in the rule name and select the custom protection object for the matching object.
4.2. In the matching condition list option, configure the matching condition of the rule. In this scenario, select the request path (Path) file extension matching content including .jpg .jpeg .webp .png .svg image static resource types.
4.3. Click +And to add a new matching condition. In the new matching condition, select the HTTP status code equal to 404 requests.
4.4. Configure the triggering method of this rule. In this scenario, configure the counting period of 10 seconds, and trigger when the count exceeds 200 times. The statistics method is based on a single client IP dimension, and is triggered when the origin server responds to the EdgeOne node. After triggering, the triggering state is maintained for 60 seconds.
4.5. Select Intercept for the execution action. The complete configuration rule is as follows:
﻿
5. After clicking OK, the rule will be deployed and take effect.
Example Scenario Three: Restricting High-Concurrency Search Engine Crawlers Access to Web Sites to Mitigate Impact on Regular Operations
A certain Y search engine provider employs a large-scale distributed crawler architecture, which lacks restrictions on access behavior. This leads to aggressive crawling activities, generating substantial traffic in a short period, potentially impacting normal operations and consuming significant resources. Therefore, rate limiting is used to identify and restrict such crawler access, mitigating its effects. For instance, the site www.example.com is affected by high-frequency visits from the Y search engine crawler. Through web security analysis, it is found that the distributed architecture used by the Y search engine crawler clusters in JA3 fingerprint and User-Agent characteristics. Hence, rate limiting rules are configured. When the number of access requests with the same JA3 fingerprint and User-Agent exceeds 60 within a 30-second statistical window, requests with identical JA3 fingerprint and User-Agent characteristics are intercepted, with the interception lasting for 10 minutes. The operational steps are as follows:
1. Log in to the Edgeone console and click Site List in the left sidebar.  In the Site List, select the Site that requires configuration to proceed to the Site Details page.
2. On the site details page, click on Security > Web Protection to navigate to the Web Protection details page. From the list of Protected domain on the left, select the domain for which you wish to enable protection, for instance: www.example.com.
﻿
﻿
3. Locate the Rate Limiting card and click on Set. This will navigate you to the Rate Limiting Configuration page. Click on Add Rule within the Custom Rate Limit Rules.
﻿
4. Within the emergent rule page, configure as per the following steps:
4.1. Enter the rule name, and select the specify object as a custom scope.
4.2. Within the selection of matching condition lists, configure the rule's matching conditions. For the current scenario, select the application layer protocol equal to HTTPS as the matching field.
4.3. Configure the trigger method for this rule. In the current scenario, set the request from the client to EdgeOne, where the JA3 fingerprint in the request feature and the User-Agent feature in the HTTP header are identical. Set the count cycle to trigger when the count exceeds 60 times within 30 seconds.
4.5. The selected action to execute is Block. The complete configuration rule is as follows:
﻿
﻿
5. After clicking OK, the rule will be deployed and activated.
Related References
When establishing rate limit rules, it is necessary to configure the rule specify scope, triggering method, and action. The explanations for each configuration item are as follows:
Note:
If your current rate rule require a match based on a known, fixed value of an HTTP header, you may configure the match object, specifying the match condition to equal the value of the designated HTTP header parameter.
If your current rate rule require a match based on a category of HTTP headers that may possess identical values, you may configure the statistical dimension, using the designated name of the HTTP header for matching.
Specify Scope
Based on the origin of the request, header characteristics, response status codes, and other factors, a combination of matching conditions 1 is established. The rate limit rule is only applied to manage the operations that meet these conditions. For more information of the matching conditions and the level of support provided by different packages, please refer to Matching Conditions.
Trigger Method
Note:
When the rate limit threshold is not met, requests will not be processed and logged.
The rule will based on the statistical rules configured in the trigger method. When the cumulative number of requests within the counting cycle exceeds the threshold, the rule is activated and executes the corresponding limiting action2. The tally is based on the technical cycle and statistical method, counting the number of requests for different feature values under the specified feature dimension (such as client IP)1. You can define the following parameters for the trigger method:
Counting Cycle: The length of the rolling time window used for counting. It supports a minimum of 1 second and a maximum of 1 hour.
Statistical Method: The method of distinguishing request sources, where the rate limit is to limit the request rate for each source. Refer to the statistical dimension for details.
Rate Threshold: The number of requests allowed per source (such as client IP) within the counting cycle.
Trigger State Retention Duration: After the rule is triggered, the duration for which requests matching the conditions of this source are continuously limited3. It supports a minimum of 1 second and a maximum of 30 days.
Statistical Dimensions
Supports statistical analysis based on one or more request characteristics. When the request features within the statistical dimension reach the rate threshold set in the trigger method, the rate limit rule is activated. You may specify the following statistical dimensions1:
Client IP: Requests originating from the same source IP will be accounted for in a singular counter. Upon exceeding the threshold, the rule's disposition action is triggered.
Client IP (prioritizing XFF header): Requests originating from the same client IP will be accounted for in a single counter, triggering the rule's disposition action upon exceeding the threshold. When the X-Forwarded-For header is present and contains a valid IP list, the first IP in the X-Forwarded-For header will be prioritized for statistics.
Designated Cookie Name: Extracts the value of the specified cookie name from the request header. Requests with identical cookie values are counted in the same counter. When the threshold is exceeded, the rule's disposition action is triggered.
For instance, when a site employs a cookie labeled user-session to mark visitation sessions, you can configure the value of the cookie named user-sessionas a statistical dimension, thereby tracking the request rate of each session. If the request rate within a single session surpass the threshold, the disposal action configured in the rule will be triggered.
Designated Name HTTP Header: Extracts the value of the specified name in the request header, with requests bearing identical header values being accounted for in the same counter. When this threshold is surpassed, the rule's disposition action is triggered. For instance, you may specify the Origin header to limit the access frequency from each external domain. When the access frequency from a particular external domain exceeds the threshold, the disposition action configured by the rule is initiated.
Specified Name URL Query Parameter: Extracts the value of the specified name parameter from the request URL query parameters. Requests with the same query parameter value are counted in the same counter, triggering the rule's disposition action when exceeding the threshold.
For instance, when a site uses a query parameter named user-session to mark access sessions, you can configure the specified name user-sessionas a statistical dimension, tallying the request rate for each session. When the request rate within a single session surpasses the threshold, it triggers the disposition action configured by the rule.
Request JA3 Fingerprint 4: Compute the JA3 fingerprint for each request, tallying the count of requests with identical JA3 fingerprints, and triggering the rule's disposition action when the threshold is exceeded. Each request corresponds to a unique JA3 fingerprint value, with no key-value model present, thus eliminating the need for specified parameter input. Considering the characteristics of JA3, it is recommended that you configure it at the same time as the User-Agent header statistics dimension to better distinguish clients.
Notes:
﻿1
: Depending on the package you subscribe to, the configurable matching conditions, statistical dimensions, and action options may vary. For more details, please refer to the Package Options Comparison.
﻿2
: If multiple rate limit rules exist, a single request can match multiple rule contents simultaneously, and the decision to trigger the rule will be based on the statistical methods of different rules. Once a rule is triggered and blocked, the remaining rules will not be triggered. When multiple rules are triggered simultaneously, they are executed in the order of priority of the triggered rules, with the rules with smaller priority values matching first. For more information, see the Web Protection Request Processing Order.
﻿3
: Once a rule is triggered, it only applies to requests that match the current rule.
﻿4
: A JA3 fingerprint is identification information formed based on the client's TLS information, which can effectively distinguish requests from different Bot networks. When a request is initiated based on a non-SSL HTTP protocol, the JA3 fingerprint of the request is empty. If you need to use a JA3 fingerprint, please ensure that the Bot management function has been enabled for your current domain.
5: If you need to perform statistics on requests with the same characteristics through a combination of multiple statistical dimensions, you need to subscribe to the EdgeOne Enterprise Edition package.
Action
When requests exceed the established threshold, corresponding restrictive actions are implemented. These include block, monitor, JSChallenge, redirect and ReturnCustomPage1. For more information, please refer to the section on Disposal Methods.
﻿