Release Notes
Announcements
Security Announcements
Computing Engine | Hit Insight Item | Description | Value Attribute |
Spark | Improper partition conditions | Improper selection of partition fields (for example, high-frequency fields or uneven data distribution) causes data skew or full table scanning when querying. | Performance optimization |
| BroadcastJoin optimization | Identify improper use of BroadcastJoin (for example, improper threshold setting), which causes tablets to perform Shuffle Join with large tables, resulting in massive network transmission and computing overhead that significantly reduces query performance. | Performance optimization Resource efficiency improvement |
| CPU resource wastage | Improper task parallelism settings (for example, too many or too few partitions), data skew, or redundant computing logic lead to underutilized or overconsumed CPU resources, resulting in resource wastage. | Performance optimization Resource efficiency improvement Cost saving |
| JOIN data inflation | Improper JOIN operations may cause data inflation, specifically: data skew, Cartesian product, and JOINs between large tables. | Performance optimization Resource efficiency improvement Cost saving |
| Data skew | Data skew (uneven data distribution) causes some partitions or tasks to process significantly larger data volumes than other partitions or tasks. Specific scenarios include JOIN key skew, GROUP BY key skew, input data skew, and Shuffle data skew. | Performance optimization Resource efficiency improvement Cost saving |
| Empty Task input | The input data of specific Tasks in the query job is empty. | Performance optimization Resource efficiency improvement Cost saving |
| ExecutorGC | Executor frequently triggers garbage collection (GC) | Performance optimization Resource efficiency improvement Stability enhancement |
| Full table scanning of partitioned tables | The query does not utilize the partition pruning feature of partitioned tables, resulting in full table scanning. | Performance optimization Resource efficiency improvement Cost saving |
| Global sorting | The query includes global sorting (for example, ORDER BY), leading to: 1. Excessive data shuffle, increasing network and disk I/O pressure. 2. Overloaded single-point sorting tasks, causing performance bottlenecks. | Performance optimization Resource efficiency improvement Cost saving |
| Run history analysis | Identify large fluctuations in duration, prolonged durations in certain Stages or Tasks, or other situations by analyzing the run history of queries. | Performance optimization Stability enhancement Resource efficiency improvement |
| Too many small input files | The input data for queries contains a large number of small files, which leads to excessive task scheduling overhead, increased metadata management pressure, and low data read efficiency. | Performance optimization Resource efficiency improvement Cost saving |
| Insufficient resources | Cluster resources (for example, CPU, memory, and disk) are insufficient to support query demands, resulting in slow task execution or failures. | Performance optimization Stability enhancement |
| Excessive data scanning | Excessive data volume scanning during queries may occur due to failing to utilize the partition pruning or index, or insufficient data filter criteria. | Performance optimization Cost saving |
| Peak memory exceeds limit | Memory usage exceeds the limit during query execution, resulting in OOM (memory overflow) or task failure. | Performance optimization Stability enhancement |
| Memory resource wastage | Due to ineffective use of cached data, improper memory allocation, and partial Executor memory limitations, memory resources are underutilized. | Resource efficiency improvement Cost saving |
| Too many small output files | Due to low storage efficiency, high NameNode load, and reduced subsequent read performance, a large number of small output files are generated from the query results. | Performance optimization Cost saving |
| Abnormal resource overhead | The resource overhead (for example, CPU, memory, and disk) for queries is abnormally high due to data skew, complex computing logic, or improper resource allocation. | Performance optimization Cost saving |
| Scheduling delay | Task scheduling delay may occur due to insufficient cluster resources, intense contention, or high scheduler load. | Performance optimization Stability enhancement |
| Scheduling skew | Uneven task scheduling may be caused by uneven data distribution or improper scheduling policy, resulting in some nodes being overloaded. | Performance optimization Resource efficiency improvement |
| ShuffleFailure | Shuffle failures can occur due to factors including excessive data volume, network or disk I/O exceeding the limit, insufficient memory, or OOM. | Performance optimization Stability enhancement |
| Slow Tasks | Some Tasks may have prolonged execution time due to data skew, complex computing logic, or insufficient resource allocation. | Performance optimization Resource efficiency improvement |
| Too many small input files | The input data for queries contains a large number of small files, which leads to the following scenarios: 1. Excessive task scheduling overhead. 2. Low data read efficiency. | Performance optimization Cost saving |
| Too small Task input data | The input data volume for specific Tasks is too small for the following reasons: 1. High proportion of task scheduling overhead. 2. Low resource utilization. | Performance optimization Resource efficiency improvement |
| StageScheduleDelay | Stage scheduling delay may occur due to incomplete preceding Stage, insufficient cluster resources, or intense contention. | Performance optimization Stability enhancement |
フィードバック