EMR by default recommends deploying in a single AZ. However, a single AZ may face risks such as complete AZ failure or insufficient resources. To enhance AZ-level reliability and resource flexibility, in Hadoop cluster scenes, you can configure a multi-AZ deployment policy. Based on deployment characteristics, the policies are divided into split policy and balanced policy.
Split policy: This policy enhances resource scalability at the AZ level by deploying services with resources from two AZs when a cluster is created. You need to configure two AZs for cluster deployment: one primary AZ and one secondary AZ. If the secondary AZ encounters an exception, the primary AZ can continue to provide services normally.
Balanced policy: By deploying across mutually backed-up AZs, the cluster can continue to operate normally even if one AZ fails. Since cross-AZ data distribution capabilities are not yet supported, AZ-level disaster recovery is currently implemented using the EMR on COS deployment scheme. This requires configuring the cluster across three AZs: one primary AZ, one secondary AZ, and one balanced AZ. The primary and secondary AZs have mutually backed-up resources, so a failure in either the primary or secondary AZ does not affect the operation of the other.
Note
Cross-AZ network incurs usage fees. Since December 2022, this has been a paid service. For more details, see VPC Purchase Guide.
Split Policy
The following document explains how to create and manage clusters based on the split policy.
Creating Clusters
Log in to the EMR Console, and on the EMR on CVM cluster list page, click Create Cluster.
1. Software configuration step: Select Hadoop Cluster Type, and select the required cluster scene. For other information, see the single cluster purchase page, and see the Creating Cluster.
2. Region and hardware configuration step: In the cross-AZ information item, select the Cross-AZ option. In the deployment policy item, select Split Policy. For the AZ information item, choose two AZs as needed, where the first row is the primary AZ, and the second row is the secondary AZ. Select subnets according to the chosen AZs.
For node configuration items, set the node specifications and quantities for each selected AZ. In the split policy, the high availability option is enabled by default and cannot be disabled.
3. For basic configuration and to confirm the configuration details, see the single-cluster purchase page. For more information, see Creating Cluster.
Cluster Scale-out
In the split policy, both the primary and secondary AZ can support on-demand scale-out of Task, Core, and Router node types.
Cluster Scale-in
When you perform scale-in under the split policy, the total number of Core node types in both the primary and secondary AZs can be reduced to a minimum of 3, with no restrictions on other node types.
Balanced Policy
The following document explains how to create and manage clusters based on the balanced policy.
Creating Clusters
Log in to the EMR Console, and on the EMR on CVM cluster list page, click Create Cluster.
1. Software configuration step: Select Hadoop Cluster Type, and select the required cluster scene. For other information, see the single cluster purchase page, and see the Creating Cluster.
2. Region and hardware configuration step: Select the Cross-AZ option for the cross-AZ information item, and choose the balanced policy for the deployment policy item. You will need to select three AZsas per your requirements. The first row represents the primary AZ, the second row the secondary AZ, and the third row the balanced AZ. Choose the subnets accordingly based on the selected AZs. For node configuration items, set the node specifications and quantities for each selected AZ. In the balanced policy, the high availability option is enabled by default and cannot be disabled.
3. For basic configuration and to confirm the configuration details, see the single-cluster purchase page. For more information, see Creating Cluster.
Viewing Cluster
After the cluster is successfully created, you can view the primary and secondary AZ information in the cluster list and on the cluster instance information page.
Log in to the EMR Console to access the cluster list page. Click the cluster ID/Name to enter the cluster instance information page:
Cluster Scale-out
The balanced policy allows for on-demand scale-out of Task, Core, and Router type nodes in the primary AZ, secondary AZ, and balanced AZ.
The balanced policy scale-out is solely for resource expansion. If the user’s cluster services are deployed with a primary/replica scheme, it is recommended to scale out resources equally between the primary and secondary AZs.
Cluster Scale-in
When you perform scale-in under the split policy, the total number of Core node types in both the primary and secondary AZs can be reduced to a minimum of 3, with no restrictions on other node types.
In the balanced AZ, full-scale scale-in is supported for Core, Task, and Router node types during the scale-in process.
