tencent cloud

RSS overview

Apache Uniffle is a unified remote shuffle service (RSS) for compute engines. It provides the ability to aggregate and store shuffle data on a remote server, thus improving the performance and reliability of large jobs.

Background

Problems with the existing shuffle solution:

• The existing shuffle solution cannot make computing serverless. Therefore, shuffle data will be computed again when server resources are preempted.
• Shuffle partitioning will lead to a large number of shuffle connections and small network packets, tending to cause timeouts in large-scale scenarios.
• The shuffle process is subject to write amplification and random I/O. In the case of a large shuffle data volume, the cluster performance and stability will be severely affected.
• The Spark shuffle service and NodeManager are in the same process, which means the NodeManager tends to restart due to a high I/O load, thereby affecting YARN scheduling.

Basic RSS characteristics

1. It stores shuffle data on a remote server and supports computing-storage separation, hybrid deployment of computing and storage, and other cluster deployment modes.
2. It supports mechanisms for aggregating and caching shuffle data, maximizes memory utilization, and reduces random access to the disk.
3. It supports multiple methods for storing shuffle data, such as MEMORY_LOCALFILE, MEMORY_HDFS, and MEMORY_LOCALFILE_HDFS.
4. It supports verifying the correctness of shuffle data, ensuring the data correctness in the computing process while filtering out invalid data.
5. It adopts the primary-secondary architecture and active-standby multi-site active-active mode to improve the cluster resource utilization and service stability.

Basic RSS architecture

Here, component features are as follows:

• Coordinator: It manages shuffle servers based on the heartbeat mechanism, stores metadata information such as shuffle server resource usage, and assigns tasks. Specifically, it assigns proper shuffle servers to Spark applications to process different partition data based on the shuffle server load.
• Shuffle server: It receives, aggregates, and writes shuffle data into the storage. It also reads shuffle data based on different storage methods (such as MEMORY_LOCALFILE).
• Shuffle client: It communicates with the coordinator and shuffle server, sends shuffle data read/write requests, and keeps the heartbeats of the application and coordinator.

RSS use

An RSS cluster provides the remote shuffle service. It cannot be used alone and needs to be associated with a Spark container cluster as follows:
Click Spark Cluster > Cluster Info > Associate RSS cluster and select a running RSS cluster.

RSS configuration

You can view the configuration files and common configuration items of different RSS roles on the Cluster Service > Configuration Management page.
The coordinator role has coordinator.conf and log4j.properties configuration files, and the shuffle server role has server.conf and log4j.properties configuration files.
We recommend you not modify common configuration items in coordinator.conf.

We recommend you not modify common configuration items in server.conf.

For more information, see incubator-uniffle.