tencent cloud


Integrating Knox with Tez

Last updated: 2022-05-16 12:52:25

    As earlier EMR versions don't come with the Timeline Server and Tomcat packages necessary for Tez UI integration, you need to complete the steps below.


    As starting Tez UI requires running Timeline Server, which uses a lot of resources, you need to assess the impact on your business and proceed with caution.

    This document describes how to integrate Knox with Tez, such as installing Tomcat and Tez UI, creating roles, configuring Timeline Server, configuring Tez, and starting services. 172.**.**.9 is the private IP of the master node, 159.**.**.70 is the public IP of the master node, and the Tez version is v0.9.2.

    Installing Tomcat and Tez UI

    cd  /usr/local/service
    wget https://jaydihu-package-1258469122.cos.ap-guangzhou.myqcloud.com/apache-tomcat-9.0.46.tar.gz
    tar -zxvf apache-tomcat-9.0.46.tar.gz
    mv /usr/local/service/apache-tomcat-9.0.46 /usr/local/service/tomcat

    Modify Tomcat port numbers:

    vim /usr/local/service/tomcat/conf/server.xml
    1. Change 8005 to 2020.
    2. Change 8080 to 2000.
    mkdir -p /usr/local/service/tomcat/webapps/tez-ui
    cp /usr/local/service/tez/tez-ui-0.9.2.war /usr/local/service/tomcat/webapps/tez-ui/
    cd /usr/local/service/tomcat/webapps/tez-ui
    unzip tez-ui-0.9.2.war
    vim ./config/configs.env

    Change localhost to the private IP of the current server.

    Creating Roles

    vim /usr/local/service/knox/conf/topologies/emr.xml 

    Modify the emr.xml file:

    1. Add the following content:
    1. Add the following content:

    Configuring YARN Timeline Server

    Modify the yarn-site.xml file in configuration management, save the changes, and restart the components whose configuration has been changed.

    Parameter Value
    yarn.timeline-service.enabled true
    yarn.timeline-service.hostname 172.**.**.9 (replace it with your IP)
    yarn.timeline-service.http-cross-origin.enabled true
    yarn.resourcemanager.system-metrics-publisher.enabled true
    yarn.timeline-service.address 172.**.**.9:10201 (replace it with your IP)
    yarn.timeline-service.webapp.address 172.**.**.9:8188 (replace it with your IP.)
    yarn.timeline-service.webapp.https.address 172.**.**.9:2191 (replace it with your IP)
    yarn.timeline-service.generic-application-history.enabled true
    yarn.timeline-service.handler-thread-count 24

    Modifying Tez Configuration

    In the tez-site.xml file, add the following parameters, save the changes, and restart the components whose configuration has been changed.

    Parameter Value
    tez.tez-ui.history-url.base http://172.**.**.9:2000/tez-ui/ (replace it with your IP)
    tez.history.logging.service.class org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService

    Starting Services

    1. Start Timeline Server.

      /usr/local/service/hadoop/sbin/yarn-daemon.sh  start timelineserver  
    2. Start Tomcat.

    3. Restart Tez.

      su hadoop
      rm -rf  /usr/local/service/knox/data/deployments/*
      /usr/local/service/knox/bin/ldap.sh stop
      /usr/local/service/knox/bin/ldap.sh start
      /usr/local/service/knox/bin/gateway.sh stop
      /usr/local/service/knox/bin/gateway.sh start
    4. Access Tez UI at the following address.


      The account and password are the same as the server login account and password.

      https://{public IP of the cluster}:30002/gateway/emr/tez/  

    Stopping Services

    If you find that the Timeline Server significantly affects your business after running it for a period of time, you can stop relevant services as follows:

    1. Stop Tomcat.
    1. Stop Timeline Server.
      /usr/local/service/hadoop/sbin/yarn-daemon.sh  stop  timelineserver
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support