Feature Introduction
The Application Insight Overview provides users with intelligent insight features for Hive queries and scheduling, simplifying Ops workflows and addressing issues of excessive resource waste. By collecting metrics from various stages of the query and job lifecycle, including submission, compilation, execution, and output, and formulating multidimensional insight policies, users can adjust resources or optimize the logic based on the insight results. The current insight feature offers comprehensive coverage at the YARN level, including TEZ/MR and various job types, while also supporting scenarios such as Hive on Yarn, Spark on Yarn, and Spark on Hive, delivering a wide range of query insights.
The primary goal of these features is to meet the market's demand for cost reduction and efficiency improvement, making resource usage more transparent and efficient.
Note:
The Application Insight Overview feature relies on the insight capabilities of YARN, Hive, and Spark engines. If activation is required, submit a ticket. Application insight policies become accessible once the feature is enabled and policy parameters can be adjusted as needed.
Operation Steps
1. Log in to the EMR console, and in the cluster list, click the ID/Name of the corresponding cluster to access the cluster details page. 2. In the cluster details page, select Insight Management > Application Insights to view insight distributions for Hive and Spark scheduling and queries, data optimization trends, and application details of exception insights along with related optimization suggestions.
3. In the cluster details, navigate to Insight Management > Application Insights > Insight Strategy tab to configure threshold values for insight item attributes based on specific business requirements.
Note:
To ensure stable operation of the cluster, the downgrade policy for application insight collection aligns with the downgrade policies for Hive queries, YARN jobs, and Spark queries. For details on Hive query insight downgrade policies, see the must-knows in Hive Query Management. For Yarn application insight downgrade policies, see the must-knows in Yarn Job Query. For Spark application insight downgrade policies, see the must-knows in Spark Query Management.