Overview
During your use of Cloud File Storage (CFS), you may need to copy data in many scenarios. This document provides recommended solutions for fast data copying in Linux operating systems.
Scenario 1: Data synchronization between CFS file systems, between different directories in a file system, and between a file system and a cloud disk
Background note: Data copying is required in many cases. The above three cases in scenario 1 are all about data copying between different directories from the perspective of a Linux operating system. Their operation methods are similar. You can use the cp command for simple and fast copying. However, if there are numerous files, the copying will be very slow due to the single-threaded running mode of cp. In this case, we recommend you use the method provided in this document for copying to achieve concurrent acceleration.
Method 1: When a detailed process report and visual UI are required, the file storage batch migration of cloud migration service CMG is recommended. Note:
Currently, cloud migration only supports batch migration from AWS file storage (EFS) to CFS. Data migration from sources like IDC, local file storage, and another CFS instance will be available soon. Stay tuned.
Method 2: When command line is required, rclone is recommended for use. See the operation instruction below for details.
Scenario 2: Data synchronization between a file system and COS
Method: You can use various basic tools provided by object storage to perform data upload and download operations. The recommended tools are as follows: COSBrowser, COSCMD. Note
This operation step is oriented to Method Two in Scenario One. For other scenarios and methods, see the above links.
Prerequisites
The source and target directories for migration already exist on the Cloud Virtual Machine (CVM).
Note:
CFS supports cross-VPC access. You can connect multiple VPCs through Cloud Connect Network (CCN) and then mount and access the file system. Directions
1. Download and install the Rclone tool.
Method 1: Download via Tencent Cloud image service (recommended)
Note:
Since it involves requesting the mirror source from the Tencent Cloud private network, only Tencent Cloud CVMs are usable. Download speed via this method is faster than direct download from the official website.
wget http://mirrors.tencentyun.com/install/cfsturbo-client//migrate_tools/rclone-v1.70.1-linux-amd64.zip && unzip rclone-v1.70.1-linux-amd64.zip && chmod 0755 ./rclone-*/rclone && cp ./rclone-*/rclone /usr/bin/ && rm -rf ./rclone-*
Method Two: Download and install via the official website link.
wget https://downloads.rclone.org/v1.70.1/rclone-v1.70.1-linux-amd64.zip --no-check-certificate && unzip rclone-v1.70.1-linux-amd64.zip && chmod 0755 ./rclone-*/rclone && cp ./rclone-*/rclone /usr/bin/ && rm -rf ./rclone-*
2. Run the following command to copy data:
rclone copy /mnt/src /mnt/dst -Pvv --transfers 32 --metadata --checkers 64 --links --create-empty-src-dirs --retries=3 --modify-window=1s
If needed, refer to the following command for backend copy:
nohup rclone copy /mnt/src /mnt/dst -vv --transfers 32 --checkers 64 --links --create-empty-src-dirs --retries 3 --modify-window=1s >> /path/to/copy.log
Note:
1. The parameters in the command are described as follows (set transfers and checkers as needed based on the system specifications):
transfers: The number of concurrent file transfers (recommended up to twice the number of cores).
checkers: The number of concurrent scans of local files (recommended up to twice the number of cores).
P: Real-time display of the progress of data copying (the progress is refreshed every 500 ms. If P is not added to the command, the progress is refreshed every minute).
links: Soft links for copying.
metadata: metadata information for copying files and directories.
vv: Print copying logs.
create-empty-src-dirs: Copy empty directories.
retries: Automatically retry failed copies (can be set as needed).
modify-window=1s: By default, rclone compares metadata such as file size, name, and mtime. If data inconsistency occurs, it proceeds with MD5 comparison. This parameter sets the mtime tolerance to 1s (default is 1ns if not set). Since the Turbo file system has second-level mtime precision, it is advisable to add this parameter when copying to it to reduce unnecessary MD5 comparisons.
2. This tool can automatically perform incremental synchronization when run repeatedly, using global scan to identify increments.
3. After data copying is complete, you can view logs to check the results for different files.