tencent cloud


Multi-Master Node Deployment

Last updated: 2021-11-18 12:47:30


    A system with a single master may be less stable, and in some online service scenarios, multiple masters are required to ensure fault tolerance. To implement fault tolerance, GooseFSRuntime provides 3 masters, which are elected using Raft, a highly consistent, decentralized, highly available distributed protocol that is widely used in engineering.

    The feature is introduced as follows.


    Before running the sample code provided in this document, install Fluid by referring to Installation and check that all the components used by Fluid are running properly.

    $ kubectl get pod -n fluid-system
    goosefsruntime-controller-5b64fdbbb-84pc6   1/1     Running   0          8h
    csi-nodeplugin-fluid-fwgjh                  2/2     Running   0          8h
    csi-nodeplugin-fluid-ll8bq                  2/2     Running   0          8h
    csi-nodeplugin-fluid-dhz7d                  2/2     Running   0          8h
    dataset-controller-5b7848dbbb-n44dj         1/1     Running   0          8h

    Normally, you shall see a pod named dataset-controller, a pod named goosefsruntime-controller, and multiple pods named csi-nodeplugin. The number of csi-nodeplugin pods depends on the number of nodes in your Kubernetes cluster.

    Setting Up an Environment

    $ mkdir <any-path>/co-locality
    $ cd <any-path>/co-locality


    Check all nodes in your cluster

    $ kubectl get nodes
    NAME                       STATUS   ROLES    AGE     VERSION   Ready    <none>   7d14h   v1.18.4-tke.13   Ready    <none>   7d14h   v1.18.4-tke.13   Ready    <none>   7d14h   v1.18.4-tke.13

    Check the Dataset resource object to be created

    apiVersion: data.fluid.io/v1alpha1
    kind: Dataset
     name: hbase
       - mountPoint: https://mirrors.tuna.tsinghua.edu.cn/apache/hbase/stable/
         name: hbase

    To facilitate testing, mountPoint is set to Web UFS in this example. If you want to mount COS, see Mounting COS (COSN) to GooseFS.

    Create the Dataset resource object

    $ kubectl create -f dataset.yaml
    dataset.data.fluid.io/hbase created

    Check the GooseFSRuntime resource object to be created

    apiVersion: data.fluid.io/v1alpha1
    kind: GooseFSRuntime
     name: hbase
     replicas: 3
         - mediumtype: HDD
           path: /mnt/disk1
           quota: 2G
           high: "0.8"
           low: "0.7"
       replicas: 3

    Specify spec.master.replicas=3 to enable the Raft 3-master mode. The parameter must be set to a positive odd number.

    Create the GooseFSRuntime resource object and check its status

    $ kubectl create -f runtime.yaml
    goosefsruntime.data.fluid.io/hbase created
    $ kubectl get pod
    NAME                          READY   STATUS    RESTARTS   AGE
    hbase-fuse-4v9mq     1/1     Running   0          84s
    hbase-fuse-5kjbj     1/1     Running   0          84s
    hbase-fuse-tp2q2     1/1     Running   0          84s
    hbase-master-0       1/1     Running   0          104s
    hbase-master-1       1/1     Running   0          102s
    hbase-master-2       1/1     Running   0          100s
    hbase-worker-cx8x7   1/1     Running   0          84s
    hbase-worker-fjsr6   1/1     Running   0          84s
    hbase-worker-fvpgc   1/1     Running   0          84s

    Check the status of the GooseFSRuntime object

    hbase   Ready           Ready            Ready     15m

    Check that all PHASE values are Ready.

    Check the Raft status (leader/follower)

    Log in to the pod of one master:

    $ kubectl exec -ti hbase-master-0 bash
    $ goosefs fs masterInfo

    You can see that the one node is leader, and the other two are follower:

    current leader master: hbase-master-0:26000
    All masters: [hbase-master-0:26000, hbase-master-1:26000, hbase-master-2:26000]
    Contact Us

    Contact our sales team or business advisors to help your business.

    Technical Support

    Open a ticket if you're looking for further assistance. Our Ticket is 7x24 avaliable.

    7x24 Phone Support