FAQs on Engines
Last updated: 2025-07-30 15:10:53
What Are the Differences Between Shared Engines and Private Engines?
Engine Type | Description | Billing Mode | Features |
Shared Engine | A public engine used by all users in the current region. | Pay-as-you-go: Billing is based on the scan volume. No fee is incurred when not in use. | 1. Configuration is not required. 2. Suitable for small-scale data or temporary data computing scenarios. |
Private Engine | Engine resources exclusively allocated to the user. | Pay-as-you-go: Billing is based on compute unit (CU) usage. The cluster can be suspended when there is no task. No fees incur during cluster suspension. | 1. Private resources with configurable resource scale and auto scaling. 2. Data computing scenarios with moderate workloads but irregular execution patterns. |
| | Monthly subscription: Billing is based on CU usage with on-demand cluster availability. | 1. Private resources with configurable resource scale and auto scaling. 2. Data computing scenarios with a large and stable workloads. |
How Many Tasks Can Run in Parallel in a Single Cluster? Can This Be Adjusted?
A single cluster supports up to 5 concurrent tasks by default. If adjustments are necessary, go to Standard Engine feature, select the target engine, and click Specification Configurations to make adjustments.
Why Is the Actual Number of CU Cores Used by A Task Less than the CU Cores Specified in the Cluster Configurations?
The following scenarios may result in fewer CU cores being utilized than configured:
1. Other tasks are running in the cluster.
2. The sum of driver and executor resources allocated for the batch job is less than the total configured cluster CU cores.
3. For pay-as-you-go clusters, resources are only allocated upon usage. When requesting large CU allocations, full capacity cannot be guaranteed.
4. The batch job is configured with enhanced networking to connect to another Virtual Private Cloud (VPC), but the target VPC lacks sufficient IP addresses to launch all executors.
Can Data Lake Compute (DLC) Clusters Access IP Addresses/Services in Other VPCs Within the Same Region? Can They Access Public Networks?
DLC engines can access same-region VPCs by creating a network configuration in Network Connection Configuration to establish connectivity with the target VPC, then specifying this configuration in batch jobs. By default, DLC engines cannot access public networks. However, if enhanced networking is configured and the target VPC has routing rules for public network access, DLC can access public network through this setup.
Will Resizing a Cluster During Task Execution Cause Task Failure?
Below is the impact of resizing across engine types during task execution:
Engine Type | Purchase Method | Impact on Task Execution |
SuperSQL-Spark job | Pay-as-you-go | No impact on tasks. |
| Monthly subscription | When the cluster is resized: Scaling out: No impact. Scaling in: Waits for pod completion before isolating or terminating resized nodes. Long-running tasks may cause operation blocking and task restart. |
SuperSQL-SparkSQL | Pay-as-you-go | Resizing the cluster will cause task restart. |
| Monthly subscription | Resizing the cluster or reducing the number of clusters will cause task restart. |
SuperSQL-Presto | Pay-as-you-go | Resizing the cluster will cause task restart. |
| Monthly subscription | Resizing the cluster or reducing the number of clusters will cause task restart. |
How to Troubleshoot Standard Engines Where Tasks Remain in Non-executing States for Extended Periods?
If your task remains in "Initializing," "Starting," or "Queued" status for an extended period, it may fail to execute normally. Follow these steps based on your task type to diagnose and resolve the issues.
1. Identifying the Task Subtype
Steps: Navigate to Historical Task Instances > Historical Task List > Task Type, Resource Group Name, and check the task type and resource group name to determine the subtype.
Task Subtype | Task Type | Resource Group Name |
SQL Task | SQL | Has a value |
Interactive SQL Task | SQL | Has no value (--) |
Spark Batch/Streaming Task | Job | Has no value (--) |
2. SQL Tasks Troubleshooting Procedure
2.1 Checking Engine and Gateway Status
1. Go to the Engine List and verify the engine status is "Running" or "Ready."
2. Check the gateway status on the management page to confirm it is "Running."
3. If the engine or gateway status is abnormal, it may indicate other processes are in progress. Please wait until the processes are complete.
4. If the status does not recover within 10 minutes, please submit a ticket to contact technical support.
2.2 Checking If Computing Resources Are in Cold Startup
1. If the task is submitted to a suspended resource group, the group may be starting (typically takes 3–5 minutes).
2. Go to the Resource Group page and confirm if the status is "Starting."
3. If startup exceeds 5 minutes, proceed with further troubleshooting.
4. If the status is neither "Running" nor "Starting," the resource group may be processing other operations. Please wait. If the problem persists for more than 10 minutes, please submit a ticket.
2.3 Verifying Resource Allocation for the Resource Group
1. If the resource group fails to start after 10 minutes, the engine may be at full capacity, preventing sufficient resources from being allocated to the group.
2. In the Resource Group List, check the resource requirements (for example, CU cores) and confirm the minimum resources required.
3. In the Engine List, click the engine name and go to "Cluster Monitoring" to review resource usage.
4. If "Occupied Cluster Capacity" reaches the total limit, insufficient remaining resources will prevent the resource group from starting.
5. Solution: Release resources by pausing or canceling other resource groups or batch/streaming job tasks.
2.4 Checking for Pending Dependent Tasks
1. By default, DLC executes multiple SQL tasks sequentially. If a preceding task is incomplete, subsequent tasks remain in "Initializing" status.
2. Confirm whether queuing is caused by unfinished dependent tasks.
2.5 Checking If the Concurrency Quantity of the Resource Group Has Been Reached
1. The default maximum concurrency quantity for a resource group is 5.
2. Tasks exceeding the concurrency quantity will remain in queued status.
3. View the concurrency quantity configuration on the Resource Group Details page, and make adjustments as needed.
4. Check the currently running task count on the Historical Task Instances page to confirm if the limit has been reached.
3. Interactive SQL Tasks (BatchSQL tasks) Troubleshooting Procedure
3.1 Checking Engine and Gateway Status
Refer to the operation steps in SQL Tasks Troubleshooting Procedure > Checking engine and gateway status.
3.2 Verifying Resource Availability
1. Check the resource usage in the "Cluster Monitoring" section of the Engine List.
2. If resources are insufficient, the task will fail to start.
3. Solution: Same as SQL tasks. Release other occupied resources.
3.3 Cold Startup Duration
Each interactive SQL task submission requires new resource allocation, with a typical cold startup duration of 3-5 minutes.
If startup exceeds 5 minutes, troubleshoot for potential issues.
4. Spark Batch/Streaming Jobs Troubleshooting Procedure
4.1 Checking Engine and Gateway Status
1. Refer to the operation steps in SQL Tasks Troubleshooting Procedure Checking Engine and Gateway Status.
2. Resource startup usually takes 3-5 minutes. If the job remains inactive beyond this period, proceed with further checks.
4.2 Verifying Engine Resource Availability
1. In the Historical Task Instances page, check the required resources (CU count for Driver and Executor).
2. Check the resource usage in the "Cluster Monitoring" section of the Engine List.
3. If resources are insufficient, the batch/streaming job will fail to start.
4. Solution: Same as SQL tasks. Release other occupied resources.
Was this page helpful?
You can also Contact Sales or Submit a Ticket for help.
Yes
No
Feedback