The resource group is a secondary queue division of the computing resources within a Standard Spark Engine. Resource groups belong to a parent Standard Engine, and resource groups under the same engine share resources with each other. The computing units (CUs) of the DLC Standard Spark Engine can be allocated to multiple resource groups as needed. You can configure each resource group's minimum and maximum CU limits, start and stop policies, concurrency, and dynamic/static parameters to efficiently manage resource isolation and workload in complex scenes such as multi-tenancy and multi-tasking.
For example, you can create separate resource groups within a Standard Spark Engine, such as a Report Resource Group, a Data Warehouse Resource Group, and a Historical Backfill Resource Group. You can set the upper and lower limits of computing units (CUs) for each resource group and assign relevant SQL tasks or jobs, such as reports and data warehouse tasks, to the appropriate resource group, ensuring resource isolation between different types of tasks and preventing individual large queries from monopolizing resources for extended periods.
Features
Resource Group Isolation
Resource groups enable resource isolation within the Standard Spark Engine. You can assign specific resource groups to different users or queries, effectively isolating resources and preventing a single user or large query from monopolizing most of the computing engine's resources.
Resource Group Elasticity
By configuring the number of Executors in a resource group for dynamic allocation, the resource group can adjust the resources used by SQL tasks or jobs based on the workload, effectively improving resource utilization.
The dynamic allocation configuration is shown in the diagram below:
Both Task 01 and Task 02 are set to dynamic allocation, each using 8 CUs at Time A. By Time B, Task 01 only requires 4 CUs, releasing 4 CUs of idle resources for Task 02 to use, thereby improving overall resource utilization. This process is illustrated in the diagram below:
Resource Group Type
Currently, DLC supports three types of resource groups: SQL analysis only resource groups, Job resource groups, and Machine Learning resource groups. It also supports default creation of "SQL analysis only" and "Job" resource groups when purchasing a standard engine, and allows users to independently create "SQL analysis only" and "Machine Learning" resource groups. The following introduces the usage scenarios and features of the three resource groups:
SQL analysis only resource group: can be set and used in modules such as Data Exploration, supporting SQL query analysis scenarios.
Job resource group: for data job scenarios. When creating a data job, select an engine to use its job resource group by default. The job's resource configuration can be set when creating the data job.
Machine Learning resource group: for AI model training scenarios using Python, ML machine learning framework, and PySpark.
Quick Configuration
Resource groups provide quick configuration options. Users can conveniently set the total specification (unit: Number of CUs) of a resource group, and the backend automatically allocates resources based on policies.
Specific policies are as follows:
1. Dynamic allocation is enabled by default.
2. Total specification [4,8): driver and executor use 2CU.
3. Total specification [8,64): driver and executor use 4CU.
4. Total specification [64,∞): driver and executor use 8CU.
Resource Group Monitoring
Resource group monitoring provides users with comprehensive task and resource usage insight to help optimize resource allocation and scheduling efficiency.
Note:
Currently only support "SQL analysis only resource group" and "Job resource group". Other type resource groups are not currently supported.
Resource Group Overview
1. Upper Resource Limit: If dynamic allocation is not enabled for the resource group, it is the fixed resource of the resource group. If dynamic allocation is enabled for the resource group, it is the Max resource set for the resource group.
2. Occupied Resources: Get the occupied resources of the "SQL analysis only resource group" and "Job resource group" in real time when pulling up Pods.
3. Remaining Available Resources: The available resources for the user in the current resource group, which is the Upper Resource Limit minus the Occupied Resources.
Detailed Metric for Resource Group Monitoring
Task metrics (Task)
Count the number of tasks in different statuses, including cancelled, failed, initialization, running, queued, and successful status.
Task duration metrics: average initialization duration, maximum initialization duration, average queue duration, and maximum queue duration. These metrics help analyze task startup and scheduling efficiency.
Resource Aggregated Metrics
Total number of Executor Cores launched by the resource group, reflecting the scale of computing resources applied for.
Total number of active Executor Cores used by the resource group, showing current actual compute resource usage volume.
Compute Unit (CU) Metrics
Resource Group CU usage, quantifying the compute unit resources consumed by the resource group.
Resource Group CU usage rate, reflecting the utilization efficiency of resource group's compute resources.
Usage Limitations
The resource group name should be globally unique. It is recommended to use an all-English name.
Terminology
Description
Illustration
(System created by default)
Exist upon engine creation, and named as default-rg-xxx.
SQL analysis only resource group: created by default when purchasing an engine and named "default-rg-xxx".
The default resource group starts in a suspended status, with settings for automatic start and automatic suspension.
The default resource group supports modification of resource configurations.
The default resource group supports configuring start/stop policies, setting concurrency limits, and adjusting dynamic/static parameters.
The default resource group supports the dependency package management function.
The default resource group cannot be deleted.
Job resource group: created by default when purchasing an engine, does not support suspend, start, or restart operations, and is named "default-job-rg-xxx".
The default resource group starts in a ready state, with automatic start and automatic suspension disabled.
The default resource group does not support modifying resource configurations and defaults to the maximum resource limit of the engine.
The default resource group does not support setting the start/stop policy and number of concurrencies, but supports setting dynamic/static parameters.
The default resource group supports the dependency package management function.
The default resource group cannot be deleted.
(User manually created)
The custom resource group supports the modification of resource configurations.
The custom resource group supports configuring start/stop policies, setting concurrency limits, and adjusting dynamic/static parameters.
The custom resource group can be deleted.
The Job resource group does not support manual creation by user or operations related to the custom resource groups.
