Hive supports using MapReduce, Tez, and Spark as execution engines, with MapReduce as the default in EMR. This document will show you how to configure the execution engine in Hive.
It is generally recommended to use the Tez execution engine instead of MapReduce for improved computation efficiency.
Development Preparation
Make sure you have activated Tencent Cloud and created an EMR cluster. For more details, see Creating a Cluster. When an EMR cluster is created, select the Hive component in the software configuration interface. If you need to switch to the Tez or Spark execution engine, check the corresponding component when creating the cluster.
Configuring Engines
Log in to the Hive client, following the directions in Hive Connection Method. The execution engine for Hive is configured using the hive.execution.engine parameter, with the default engine being mr. You can use the following command to check the current execution engine: hive> set hive.execution.engine;
hive.execution.engine=mr
Switching Execution Engine in the Command Line
Use the following command to modify the execution engine for the current client session:
Modify the execution engine to Tez:
hive> set hive.execution.engine=tez;
Modify the execution engine to Spark:
hive> set hive.execution.engine=spark;
After execution, you can enter the following command to verify:
hive> set hive.execution.engine;
If the returned information matches the execution engine you set, it indicates that the configuration is successful.
Note:
The above execution engine switch operation is only effective for the current client session; the default execution engine will be used when accessing by other clients or after disconnecting and reconnecting.
Switching Execution Engine in the Configuration File
You can globally switch the default execution engine through the EMR console by navigating to Cluster Services > HIVE > Configuration Management. Locate the hive-site.xml configuration file and set the following parameter value:
Modify the execution engine to Tez:
hive.execution.engine = tez
Modify the execution engine to Spark:
hive.execution.engine = sparkP
Perform Save Configuration > Save and Deploy, and Restart HiveServer2. After the restart is completed, the default execution engine of Hive will be modified.