This document describes how to deploy and use TACO-Training on GPU instances.
Notes
Currently, TACO-Training is supported only by Cloud GPU Service.
Currently, the three acceleration components of TACO-Training have been integrated into the same Docker image, which can be pulled at the following address:
1. Create at least two instances meeting the following requirements as instructed in Purchasing NVIDIA GPU Instance:
Instance: We recommend you select the Computing GT4 GT4.41XLARGE948 8-card model.
Image: Select CentOS 7.8 or Ubuntu 18.04 or later and select Automatically install GPU driver on the backend to use the automatic installation feature to install the GPU driver.
Automatic installation of CUDA and cuDNN is not required for this deployment, and you can choose as needed.
System disk: We recommend you configure a system disk of 100 GB or above in size to store the Docker image and intermediate state files generated during training.
2. Install Docker as instructed in the corresponding document based on the operating system type of your instance:
docker run --network host -it --rm --gpus all --privileged --shm-size=32g --ulimit memlock=-1 --ulimit stack=67108864 --name lightcc ccr.ccs.tencentyun.com/qcloud/taco-training:cu112-cudnn81-py3-0.3.2
3. Run the following command to view the LightCC version:
pip show light-horovod
Note:
When the kernel protocol stack is used for NCCL network communication, or if the runtime environment of the HARP protocol stack is not configured, you need to move the /usr/lib/x86_64-linux-gnu/libnccl-net.so file in the image to a path other than the system lib directory, such as the /root directory, as the system will check whether the HARP configuration file exists in a certain directory in the lib directory during init and will report an error if the file doesn't exist.
Environment variable configuration
LightCC environment variables are as detailed below, which can be configured as needed:
Environment Variable
Default Value
Description
LIGHT_2D_ALLREDUCE
0
Whether to use the 2D-Allreduce algorithm
LIGHT_INTRA_SIZE
8
Number of GPUs in a 2D-Allreduce group
LIGHT_HIERARCHICAL_THRESHOLD
1073741824
Threshold for 2D-Allreduce in bytes. Only data of a size less than this threshold can use 2D-Allreduce.
LIGHT_TOPK_ALLREDUCE
0
Whether to use TOPK to compress the communication data
LIGHT_TOPK_RATIO
0.01
Compression ratio of TOPK
LIGHT_TOPK_THRESHOLD
1048576
Threshold for TOPK compression in bytes. Only communication data of a size greater than or equal to this threshold can be compressed through TOPK.
LIGHT_TOPK_FP16
0
Whether to convert the values of the compressed communication data to FP16
Demo
1. After creating two GPU instances, install LightCC and configure environment variables in the above steps.
2. Run the following commands in the container to configure passwordless SSH login for the instances:
# Allow the root user to use the SSH service and start the service (default port: 22)
sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
service ssh start && netstat -tulpn
# Change the default SSH port in the container to 2222 to avoid conflicts with the host
sed -i 's/#Port 22/Port 2222/' /etc/ssh/sshd_config
service ssh restart && netstat -tulpn
# Set `root passwd`
passwd root
# Generate an SSH key
ssh-keygen
# Configure SSH to use port 2222 by default
# Create `~/.ssh/config`, add the following content, and save and exit the file:
# Note: The IP used here is the IP displayed in `ifconfig eth0` of the two instances.
Host gpu1
hostname 10.0.2.8
port 2222
Host gpu2
hostname 10.0.2.9
port 2222
# Configure mutual passwordless login for the instances and local passwordless login for each instance
ssh-copy-id gpu1
ssh-copy-id gpu2
# Test whether passwordless login is configured successfully
ssh gpu1
ssh gpu2
3. Run the following command to download the benchmark test script of Horovod:
Here, the command parameters are used for an 8-card model. To configure another model, modify the -np and -H parameters. Other parameters are as detailed below:
NCCL_ALGO=RING: Select the ring algorithm as the communication algorithm in NCCL.
NCCL_DEBUG=INFO: Enable debugging output in NCCL.
-mca btl_tcp_if_include eth0: Select the eth0 device as network device for MPI multi-server communication. As some ENIs cannot communicate, you need to specify the ENI if there are multiple ones; otherwise, an error will occur if MPI chooses an ENI that cannot communicate.
The reference throughput rates of LightCC multi-server training benchmark in two GT4.41XLARGE948 instances are as detailed below:
If the input result contains "Set up HARP successfully", and the ztcp*.conf configuration file is generated in the /usr/local/tfabric/tools/config directory, the configuration has been completed successfully.
Installation
1. Run the following command as the root user and install HARP through a Docker image:
Here, the command parameters are used for an 8-card model. To configure another model, modify the -np and -H parameters. Other parameters are as detailed below:
NCCL_ALGO=RING: Select the ring algorithm as the communication algorithm in NCCL.
NCCL_DEBUG=INFO: Enable debugging output in NCCL. After it is enabled, HARP will output the following content:
-mca btl_tcp_if_include eth0: Select the eth0 device as network device for MPI multi-server communication. As some ENIs cannot communicate, you need to specify the ENI if there are multiple ones; otherwise, an error will occur if MPI chooses an ENI that cannot communicate.
After NCCL initialization, you can view the network output:
5. HARP is integrated to NCCL as a plugin and is enabled automatically without any configuration required. To disable HARP, run the following command in the container:
Yes
No
Was this page helpful?