Alluxio users often have existing applications accessing their existing storage systems (Under-FileSystem). Alluxio can be added to existing ecosystems, but one thing that must always be changed is the URI used by the application. The Transparent-URI feature allows users to access their existing storage systems without changing URIs at the application level.
- You need to restart the Alluxio service after modifying the configuration in
- Once ShimFileSystem is configured, master will need to route URIs that are native to external storage system, to Alluxio namespace. This requires COSN to have been mounted to Alluxio namespace.
- To disable the Transparent-URI feature, roll back the
fs.cosn.implconfiguration item in
mount command is one of Alluxio's most distinctive commands. It is similar to Linux's
mount command, through which users can load disks, SSDs and other storage devices to the local file system of Linux. In Alluxio, the concept of mounting is further extended to the distributed system level, that is, users can mount one or more other storage systems/cloud storage services (such as HDFS and COS) to the Alluxio distributed file system using the
mount command. So they can run distributed applications on Alluxio, such as Spark, Presto, and MapReduce, without any adaptation or even knowledge of the specific data access protocol and path. Users only need to know the path corresponding to the data in the Alluxio file system. It greatly facilitates application development and maintenance.
Beginning from EMR-Alluxio v2.5.1, Alluxio's UFS supports the COSN protocol. COS UFS has relatively poor read/write performance and is unstable. In order to solve this issue, the community provides the underlying COSN UFS. Both COS UFS and COSN UFS are used to access Tencent Cloud COS. COSN UFS is a great improvement compared with COS UFS. Its read/write performance are twice as good as that of COS UFS, and it has better stability. Therefore, COSN UFS is strongly recommended. The maintenance for COS UFS will stop after EMR-Alluxio v2.6.0.
Example of mounting COSN:
alluxio fs mount --option fs.cosn.userinfo.secretId=xx \ --option fs.cosn.userinfo.secretKey=xx \ --option fs.cosn.bucket.region=ap-xx \ --option fs.cosn.impl=org.apache.hadoop.fs.cosnative.NativeCosFileSystem \ --option fs.AbstractFileSystem.cosn.impl=org.apache.hadoop.fs.CosN \ --option fs.cosn.userinfo.appid=xx \ /cosn cosn://COS_BUCKET/path
Configure the COS information in each
|fs.cosn.userinfo.secretId||COS secret ID|
|fs.cosn.userinfo.secretKey||COS secret key|
|fs.cosn.bucket.region cos region||Region name such as
|fs.cosn.userinfo.appid||The AppID of root account|
|COS_BUCKET COS BUCKET||Bucket name without the AppID suffix|