Background
Tencent Cloud Smart Advisor-Chaotic Fault Generator provides fault actions for simulating a primary-secondary switch scene in the TencentDB for TDSQL-C to verify the overall high availability of TDSQL-C in your business.
Primary-secondary switch experiments are intended to help developers in system tests and experiments in a more complex and realistic environment so that possible problems and risks can be identified. Through experiments and tests in chaos engineering, developers can have a more comprehensive understanding of system operating modes and performance characteristics so that they can develop countermeasures and policies for different fault scenes to improving system stability and availability.
Note:
This fault action only supports injection of primary-secondary switch faults into cross-availability zone instances. If an instance deployed in a single availability zone is selected, injection will fail and a prompt will be given in environmental check. Corresponding instances can be selected through TDSQL-C Console to adjust deployment mode. Example of Instance Deployment Adjustment:
1. Log in to TDSQL-C Console, click Cluster List, and select the instance in a single AZ. 2. Click Edit Icon in Deployment Method in instance information.
3. Select Multiple Availability Zones (Yes), complete multi-availability zone configuration, and click Confirm.
Experiment Implementation
Step 1: Experiment Preparation
A cross-availability zone TDSQL-C instance. Pay attention to availability zones where primary and secondary nodes are located.
Step 2: Experiment Orchestration
2. Click Skip and create a blank experiment.
3. Fill in experiment information and select Object Type TDSQL-C.
4. Add instances and add fault actions.
5. Add Primary-secondary switch experiment actions. No parameter is to be configured for this fault. Click Next.
6. Click Next, configure the Experiment Orchestration method and optionally configure Guardrail Policy Monitoring Metrics, and then click Submit to create an experiment.
Step 3: Experiment Execution
During fault execution, primary-secondary switch in TDSQL-C instance will be triggered. Changes in primary and secondary node architectures can be observed through Cloud Database TDSQL-C Console. Go to Experiment Details, click Execute to start executing an experiment.
During fault injection, state of switch can be observed through TDSQL-C Console.
After a successful fault execution, you can observe changes in the primary node through Cloud Database TDSQL-C Console, and the primary node has been switched from the original primary availability zone to cross-availability zone replica availability zone node. Execute fault recovery actions to recover instance deployment to the pre-fault state.
After a successful recovery, it can be seen through the console that the node has been recovered to the pre-fault state.