Overview
Provides a quick overview of detailed metrics such as the user submitting Spark jobs, job status, and duration. It also supports historical job comparison, job insights, and task execution information at the job level.
Directions
1. Log in to the EMR console, and click the corresponding Cluster ID/Name in the cluster list to enter the cluster details page. 2. On the cluster details page, select Job Management > Spark to view query management, access information on related Spark jobs, task details, application execution result insights, and application monitoring comparison.
At the job level, it provides multi-dimensional filtering options such as user, application name, queue name, job type, duration, and related throughput resources.
Note
For the new features of task information, application insights, and application comparison for Spark-type applications, you need to check whether the Spark History version is compatible. Use the following command curl https://localhost:10000/api/v1/applications" | json_pp to check. If the returned data is not in a normal JSON format, then the Spark History version is not compatible, and you may submit a ticket to request to enable the relevant features. Job queries collect ResourceManager data every 30 seconds. The impact of the collection operation on cluster operations is minimal and can be ignored.
3. In the job list, click More > Application Insights to view detailed insight items for the application, along with related insight rules, results, and suggestions.
4. In the job list, click More > Task Information to view the job’s task list and the task’s running logs.