Background
E-commerce, social media, online gaming, and streaming services typically need to handle massive user traffic, which places significant stress on servers handling request loads and data read/write operations. For example, during promotional events in the e-commerce industry, a large number of users simultaneously access e-commerce websites. If the websites cannot handle such high traffic, it may lead to website crashes, loss of order data, user attrition, and other issues, resulting in substantial financial losses for enterprises.
Based on the above context, TSA-Chaotic Fault Generator (TSA-CFG) integrates the capabilities of the Performance Testing Service (PTS) platform, providing core capabilities such as orchestration, massive-scale stress testing, one-click automated resource preparation, and real-time system assessment in stress testing scenarios. This enables users to more flexibly implement stress testing within CFG and detect critical thresholds for service stability.
GameDay
Experiment Objective
Take the launch of a new game server as an example. To ensure normal business operations, users need to conduct several stress tests in advance. By observing and assessing steady-state metrics such as server response time, throughput, and concurrent connections under different gamer behaviors, users can determine whether the system stability meets requirements.
Experiment Design
Based on the characteristics of the game and gamer behavior patterns, develop a series of test cases including login, character creation, in-game quests, and trading operations, and initiate requests at different gamer volumes. Then, under both regular gamer behavior traffic and peak traffic, observe the system response for the following test items to validate system stability and reliability. Common test items for reference include:
Load testing: It measures the changes in various system performance metrics as the load gradually increases.
Stress testing: It verifies the maximum level of service a system can provide under specified performance bottleneck thresholds.
Concurrency testing: It simulates concurrent user access to the same software, module, or data record to test the software's concurrency performance and whether issues like deadlocks occur.
Burst testing: It tests whether services can be consistently and stably provided when services are deployed across regions or AZs during force majeure or other unexpected incidents that cause service resources to crash and become unavailable.
For a typical business architecture where HTTP requests are sent to the web services and the web services read and write data, this document focuses on demonstrating a CFG scheme for testing the stress-bearing capacity of the web services and database components.
Experiment Execution
Step 1: Creating a Test Scenario
1. Log in to the PTS console. 2. Create a PTS project.
3. Create a testing scenario in script mode, configure key parameters such as number of concurrent users, stress testing duration, and stress testing script. For details, see Quick Start. Script reference 1: PTS is compatible with JavaScript scripts. For example, taking HTTP requests as an example, the JavaScript script is as follows. For more stress testing scripts, see the common script template examples on the right side of the script editor in the PTS console.
import http from 'pts/http';
import { check } from 'pts';
export default function () {
const resp = http.post(
'http://game.demo/create_character',
{
name: 'user001',
profession: 'zhanshi'
},
{
headers: {
'Content-Type': 'application/json',
},
}
);
}
Script reference 2: Web service demo.
from flask import Flask, request, jsonifyimport
pymysqlapp = Flask(__name__)
def create_conn():
conn = pymysql.connect(host='10.0.0.1',
user='root',
password='password',
db='mydb',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor)
return conn
@app.route('/create_character', methods=['POST'])
def create_user():
data = request.get_json()
name = data.get('name')
profession = data.get('profession')
conn = create_conn()
try:
with conn.cursor() as cursor:
sql = "INSERT INTO `character` (`name`, `profession`) VALUES (%s, %s)"
cursor.execute(sql, (username, password))
conn.commit()
finally:
conn.close()
return jsonify({'status': 'success'}), 200
Script reference 3: Reduce the request links and only test the critical values of database reads/writes.
import sql from 'pts/sql';
const db = new sql.Database(sql.MySQL, "user:passwd@tcp(ip:port)/database")
export default function () {
const randomName = 'zhangsan_' + Math.random().toString(36).substring(2, 8);
const sql = `insert into \\`character\\`(profession,name) values ('zhanshi', '${randomName}')`;
let result = db.exec(sql)
console.log(JSON.stringify(result));
let rows = db.query("SELECT * FROM `character` order by id desc limit 100");
console.log(JSON.stringify(rows));
}
Step 2: Creating an Experiment
2. Click Create Experiment, enter the basic information about the experiment, and click Next.
3. Choose Others > PTS from the Experiment Instance drop-down list, click Add via Search, and select the created stress testing scenario based on the project.
4. After the instance is added, click Add Action, select Execute PTS Task as the experiment action, and click Next.
5. Add a new action group, choose Database > MySQL, and add an instance.
6. Select a fault action: Click MySQL Primary Node Fault to simulate the impact on service throughput and response rate when a node fails.
7. Set Execution Mode and Guardrail Policy, and add metrics for Observability Metrics in the Global Configuration section. After the configuration is complete, click Submit to complete the experiment creation.
Step 3: Executing the Experiment
1. Go to the experiment details panel, and click Execute in the fault action group or Start Experiment in the lower part of the panel to inject a fault. The stress testing action starts first.
2. Observe the information on the Stress Testing Feedback tab in real time.
Observe steady-state metrics and monitoring performance, locate potential business issues through logs, and resolve the slow response issue in high-concurrency scenarios.
3. Others: Simulate normal traffic through automated testing to verify the capabilities of other cloud services.