Feature Introduction
Provides data distribution and trend information based on the storage volume of databases and data tables.
Provides the distribution of data tables based on their last access time, serving as a reference for hot and cold data distribution.
The proportion of small files at the data table level and the data volume at the partition level within tables can help identify issues such as small files and partition data skew.
Directions
1. Log in to the EMR console, and click the corresponding Cluster ID/Name in the cluster list to enter the cluster details page. 2. On the cluster details page, click Cluster Services, then select Hive component and select Operations > Data Table Analysis in the top-right corner. This provides the relevant data table and data information collected based on the Hive MetaStore up to the last collection point.
3. Statistical Graph
3.1 Statistics view
View metrics such as the Hive database, total number of tables, total storage volume, and daily increase and daily comparison information for related metrics.
3.2 See the last access time of data tables to view the distribution, which can be used as a reference for hot and cold data distribution.
Note:
|
Within 3 months | Most recent access time less than 3 months |
3 months to 1 year | Most recent access time greater than or equal to 3 months and less than or equal to 1 year |
1 year to 5 years | Most recent access time greater than 1 year and less than or equal to 5 years |
Over 5 years | Most recent access time greater than 5 years |
Others | Collection not enabled at the COS/CHDFS side Most recent access time for stored data not collected |
4. The trend view provides the historical growth trend of the number of databases, tables, and table storage size. It includes a dimension for tables with small files, allowing observation of the distribution and growth of small file tables.
5. You can view the database to which a table belongs, storage size, number of files, proportion of small files, and the table’s partition details. The cloud file storage size and the proportion of small files provide an intuitive view of Hive small file issues.
6. Click Operation to view partitions, where you can see information such as partition name, partition size, and total number of files. Partition size and total number of files help identify partition skew and file quantity details.
Risk note
The analysis data for data table analysis will be collected daily starting at 14:00 UTC+8.
1. It mainly collects HMS database and table, partition metadata, and NameNode directory information. This operation will slightly increase the request load for HMS and NameNode. If the request volume continuously exceeds the load capacity, you can ticket feedback to disable this feature. 2. HMS and NameNode data collection only involves metadata and does not affect specific business data.