During your use of Cloud File Storage (CFS), you may need to copy data in many scenarios. This document provides recommended solutions for fast data copying in Linux operating systems.
Scenario 1: Data synchronization between CFS file systems, between different directories in a file system, and between a file system and a cloud disk
Data copying is required in many cases. The above three cases in scenario 1 are all about data copying between different directories from the perspective of a Linux operating system. Their operation methods are similar. You can use the
cp command for simple and fast copying. However, if there are numerous files, the copying will be very slow due to the single-threaded running mode of
cp. In this case, we recommend you use the method provided in this document for copying to achieve concurrent acceleration.
Scenario 2: Data synchronization between a file system and COS
When the public network is required for storage access or data import, it is recommended to use COS as a transit. You can use various basic tools provided by COS for data uploading and downloading. The COSBrowser and COSCMD tools are recommended.
The directions apply to scenario 1. For scenario 2, refer to COSBrowser and COSCMD.
The source and destination directories for copying exist on the CVM.
CFS supports cross-VPC access. You can connect multiple VPCs through Cloud Connect Network (CCN) and then mount and access a file system.
1. Download and install the Rclone tool.
wget https://downloads.rclone.org/v1.53.4/rclone-v1.53.4-linux-amd64.zip --no-check-certificate
chmod 0755 ./rclone-*/rclone
cp ./rclone-*/rclone /usr/bin/
rm -rf ./rclone-*
2. Run the following command to copy data:
rclone copy /mnt/src /mnt/dst -Pvv --transfers 32 --checkers 64 --copy-links --create-empty-src-dirs
This tool does not copy metadata information such as owner, ctime, and atime, by default. If you need to copy metadata information, use the
The parameters in the command are described as follows (set
checkers as needed based on the system specifications):
transfers: The number of concurrent file transfers (recommended up to twice the number of cores).
checkers: The number of concurrent scans of local files (recommended up to twice the number of cores).
P: Real-time display of the progress of data copying (the progress is refreshed every 500 ms. If
P is not added to the command, the progress is refreshed every minute).
copy-links: Soft links for copying.
vv: Print copying logs.
create-empty-src-dirs: Copy empty directories.
3. After data copying is complete, you can view logs to check the results for different files.