Background
Huanle Entertainment (Shanghai) Technology Co., Ltd. (hereinafter referred to as "Huanle Entertainment") is a global game development and publishing company, focusing on the following two major genres: massively multiplayer online role-playing games (MMORPGs) and massively multiplayer online action games (MMOACTs). It has successfully launched many well-known games such as Arcade Three Kingdoms, Dragon Nest, and Hero Duel. In April 2023, Huanle Entertainment planned to launch its blockbuster new game Ragnarok Online (RO) in Southeast Asia, which fully used the cloud architecture and services at Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) layers of Tencent Cloud. The project has a large service scale and affects a wide range of users. To ensure the smooth launch of the new game, Huanle Entertainment collaborated with the TSA-Chaotic Fault Generator (TSA-CFG) team to conduct stability experiments for three core business scenarios.
Business Challenges
1. High Concurrency Pressure
The gaming business needs to support massive simultaneous access from a large number of online players, especially in scenarios such as new game launches, events, and unexpected incidents. This often imposes high concurrency pressure on servers, networks, and database resources. The system must have auto scaling capabilities to adjust resource allocation in real time to ensure game availability.
2. High Network Stability Requirements
The gaming service highly relies on real-time interactive experiences, requiring network resources to provide low latency and high reliability. It must maintain stable connections across diverse hardware and network conditions.
3. Complex Disaster Recovery Scenarios
The gaming industry imposes strict requirements for high availability and business continuity. Cloud services need to deliver robust disaster recovery capabilities to address faults, enable rapid recovery, and ensure the gaming business stability.
Solutions
1. Custom Solutions for Gaming Scenarios
The TSA-CFG expert team collaborated with the high availability service team to develop three fault simulation scenarios for the gaming business, including the load balancing disaster recovery capability of the access layer, virtual machine disaster recovery capability of the logic layer, and cloud database disaster recovery capability of the data layer. With rich internal practical experience and profound industry insights, TSA-CFG provides targeted fault scenario solutions and fault action orchestration to achieve low-cost and high-efficiency chaos experiments.
2. Fault Recovery Mechanism and Monitoring Metric System
TSA-CFG integrates the monitoring metrics systems of various basic cloud products. When a fault occurs, users can leverage the fault recovery mechanism to promptly switch to backup resources to ensure the normal operation of the gaming business. Simultaneously, implementation personnel can use information such as logs, monitoring data of business metrics, Ops alarm messages, and business performance data on the product page to comprehensively observe the actual performance of the system after a fault injection to ensure that the system's availability and reliability meet expectations.
3. Abundant Cloud Fault Scenarios and Automated Execution Capabilities
The platform supports fault injections for over 20 types of object resources such as hosts, containers, databases, and DC resources of Tencent Cloud, providing nearly 100 fault scenarios. It can simulate more realistic fault scenarios for the gaming business requirements to test system stability. The fault action orchestration on the platform is easy to operate, reducing communication and Ops costs, and significantly improving the overall chaos experiment efficiency.
Customer Benefits
1. Enhanced disaster recovery capabilities: The system availability and reliability are comprehensively verified. This helps improve the speed of fault detection, troubleshooting, and recovery, reduce risks, and ensure significant improvements in system stability and reliability.
2. In-depth system insights: In practice, the business Ops team has a deeper understanding of the disaster recovery capabilities of key services and the system switching process in the event of a fault. This facilitates problem solving and enhances confidence in product stability and availability.
3. Stable launch of new games: During the new game launch experiment, Huanle Entertainment successfully addressed a series of disaster recovery challenges for key businesses, ensuring the smooth launch of the new game in Southeast Asia. In addition, it won the first place in the best-selling list in five countries on the first day of launch.