tencent cloud

Overview

Fluid is an open-source Kubernetes-native distributed dataset orchestrator and accelerator for data-intensive applications, such as big data and AI. It is hosted by the Cloud Native Computing Foundation (CNCF) as a sandbox project. By defining the abstraction of dataset resources, it features:

• Native support for dataset abstraction: Implements the basic capabilities required for data-intensive applications to achieve efficient data access and reduce the cost of multidimensional management.
• Cloud data prefetch and acceleration: Fluid provides data prefetch and acceleration for cloud applications by using a distributed cache engine (GooseFS/Alluxio) with data observability, portability, and horizontal scalability.
• Co-orchestration for data and applications: During application and data scheduling on the cloud, it takes their characteristics and location into consideration to improve the performance.
• Multi-namespace management support: Allows you to create and manage datasets in different namespaces.
• Heterogeneous data source management: Unifies the access to underlying data from different sources (COS, HDFS, and Ceph), applicable to hybrid cloud use cases.

Key Concepts

Dataset: A dataset is a set of logically related data that can be used by computing engines, such as Spark for big data and TensorFlow for AI. Smart data applications create core industry values. Managing datasets may require features in different dimensions, such as security, version management, and data acceleration.

Runtime: The execution engine that enforces dataset security and provides version management and data acceleration capabilities. It defines a set of APIs for dataset management and acceleration throughout the lifecycle.

GooseFS Runtime: It is a Java-based implementation of the execution engine developed by Tencent Cloud's COS team, supporting dataset management, caching, and COS. GooseFS is a Tencent Cloud product with dedicated product-level support, but its code is not open-source. Fluid enables dataset visualization, elastic scaling, and data migration by managing and scheduling GooseFS Runtime.

Alluxio Runtime: Based on open-source Alluxio, it is an implementation of the execution engine for dataset management and caching, supporting PVC, Ceph, and CPFS computing, thereby effectively supporting hybrid cloud use cases. Alluxio is an open-source scheme. In spite of the joint efforts of Tencent Cloud and the community to promote the stability and performance of its data caching, there will be a delay in timeliness and response. Fluid enables dataset visualization, elastic scaling, and data migration by managing and scheduling Alluxio Runtime.

Add-on Installation

Prerequisite dependencies

• Kubernetes cluster (v1.14 or later)
• CSI support in the cluster

Parameter configuration

During Helm deployment, all configuration items are included in values.yaml.
Some fields may need to be customized, as listed below:

Best Practices

For more information, see the Fluid documentation.