Background
TencentDB for Redis allows automatic read-write separation through the Proxy layer. Proxy node can determine whether replica read-only mode is to be enabled. Proxy layer acts as a proxy for Redis sharded cluster. A fault in Proxy layer will have direct impact on access to data in TencentDB for Redis. Users can use the TencentDB for Redis Proxy node fault action provided by the platform to check if a fault in Proxy layer node will have impact on their businesses.
Experiment Implementation
Step 1: Experiment Instance Preparation
Select one or more TencentDB for Redis instances as instance objects of an experiment.
Step 2: Experiment Orchestration
2. Click Skip and create a blank experiment. Fill in experiment information, select Redis Memory Edition under Cloud Resources for Resource Object, add target instance.
3. Click Add Now to add an experiment action, select Redis Proxy node failure , and click Next.
4. Fill in corresponding action parameters, and click Confirm.
Note:
There are two action parameters required for this action: fault injection availability zone and fault node proportion (%).
Fault Injection Availability Zone:
Primary Availability Zone: This means that a Proxy node where a fault is to be injected will be selected from primary availability zone.
Secondary Availability Zone: This means that a Proxy node where a fault is to be injected will be selected from secondary availability zone.
All Availability Zones: This means that a Proxy node where a fault is to be injected will be selected from all availability zones.
Faulty Node Ratio (%) : This parameter is an integer ranging 1~100, indicating that a certain number of proxy nodes will be selected for fault injection.
For example: Parameters for a certain experiment can be so configured that fault is injected into 50% of Proxy nodes in the primary availability zone. If the total number of proxy nodes in the primary availability zone in the instance is five, 5 * 50% = 3 (rounded up) nodes will be randomly selected from the 5 proxy nodes in the primary availability zone for fault injection.
5. After the configuration of action parameters, click Next. After confirming all configurations, click Submit to complete experiment creation.
Step 3: Experiment Execution
1. Initial state of a TencentDB for Redis Proxy node. You can go to TencentDB for Redis console, click corresponding instances and select System Monitoring > Monitoring Metrics > Instance Monitoring > Proxy Monitoring. 2. To execute an experiment, click Action Card and observe the results of execution.
It can seen that a fault has been injected into the specified node and the node has crashed. And then there will be a new Proxy node.
[2023-02-02 10:23:10]: General Parameters Configuration:
[2023-02-02 10:23:10]: Lead waiting time (s): 0
[2023-02-02 10:23:10]: Action timeout (s): 1800
[2023-02-02 10:23:10]: Post waiting time (s): 0
[2023-02-02 10:23:10]: Custom parameters configuration:
[2023-02-02 10:23:10]: Fault node proportion (%): 60
[2023-02-02 10:23:10]: Fault injection availability zone: primary availability zone
[2023-02-02 10:23:10]: Start execution [Redis Proxy Node Fault] action, instances involved: ['crs-xxxxxxx']
[2023-02-02 10:23:11]: [Redis Proxy node fault] action has been executed, and monitoring on returned results of action is started.
[2023-02-02 10:23:14]: [Redis Proxy Node Fault] action call [Successful], details:
Instance: [crs-xxxxxxx] Execution result: [Successful] Details: Instance: crs-xxxxxxxx, fault injection into CRS Proxy node is successful, injected Proxy node object ID = ['d06d2ffb1b600ffa877137c1c5c5265221cd55fa', '1ce5fec58407759a9ff04c30c1a9e485911ac25d']